Okay, you just need to remember one thing, and that is that however much you connect with him, he isn't the only bloke in the world that can make you happy.
There are loads of men that can do that, it's just finding them, or meeting them. To choose him is unfair on him, his wife and their baby - and I am saying this from the point of view of someone very attracted to a married bloke at the moment, but not even considering doing anything about it.
Why? Because no matter how well we get on, I have got on with other blokes like this in the past, and I will again, if I meet enough people. There's no justification for choosing this one, and he is promised to someone else. Sure if one day he contacts you again and says he is single, then by all means.
The main problem is isolation, loneliness and you sound so sad, so upset. Being with your DH is stopping you from meeting other people you might hit it off with, perhaps.
And you have a responsibility really to sort out your marriage, one way or the other, before you involve someone else.
Good luck, it must be bringing up some very painful feelings for you x