I'm sorry to be blunt but I stand by my feelings that you should end your marriage if that's what you intend to do before even thinking about acting on any feelings that you have for anyone else.
You've made a committment to your husband and you say he's taken responsiblity for your marriage problems and wants to try to make things work.
On top of that you have a baby on the way.
In my eyes there is no dilema here.
I agree with what other posters have said about perhaps not moving away right now, and how difficult being isolated will be, but if you've agreed to it and it's too late to change, then what have you got to lose in giving it your best shot?
I also agree that this man does not sound like a good replacement. To go through with his wedding, when he'd admitted he had strong feelings for you, was selfish and weak. If he really felt that strongly about you he should never have got married. There is no excuse, it can be called off right up until the moment you say 'I do'. How could you ever fully trust him??
The other man is giving you what you lack in your marriage, the excitement and variety, with none of the mundane things that we all know come with a long relationship.
But you CAN get everything you need from your husband - if you love him, if you still like him, and if you both really want to try.
Please try to find the strength to cut off all contact with this other man once you move. You owe it to yourself, your family, and your relationship, to find the strength that you lost when you decided to go and sleep with someone else.
I would assume you wouldn't even contemplate going off and sleeping with this man whilst pregnant with your husband's baby anyway? So there really is no point in keeping in touch with him or seeing him.
If you give you marriage everything you've got (and your husband does too), and it doesn't work out. End it and walk away.
Only then should you get back in touch with the other man. If it's meant to be then it's meant to be. But to be honest I think it sounds like you'd be better off without him.
I'd change your e-mail and phone nuber and get him to do the same too. Remove all temptation. Then in any moments of weakness or if you're feeling a bit lonely/fed up, you don't even have the option to contact him.
Find some friends (on MN and in RL) and use them to support you when you need cheering up or need a different perspective.
You definitely don't need this other man! I promise!
[sorry so long - subject close to my heart!]