Maybe the connection was entirely genuine. I certainly wouldn’t automatically dismiss it as a tissue of lies and future-faking because he didn’t end his marriage. You say yourself you’re unsure whether you would have wanted that, anyway. The intensity of snatched moments and secrecy is no guide to whether you would have gone on to have a happy relationship once there were no barriers to you being together, and you were dealing with the reality of living together and bringing up your young child. Strong connections can be situational, like friendships— their impermanence is doesn’t make them less real, it just means that the needs of two people were being met by a specific conjunction of circumstances at one point.
I’ve noticed with two male friends who ended their apparently happy marriages because of affairs, and who went on to marry/longterm cohabit with their affair partner, that in fact they’re leading striking similar lives in their subsequent relationships.
One ran a very successful business with his first wife and complained after the marriage ended that he hadn’t been happy for years because he was leading her life, the business was her vision. Yet now he’s running a new business with his second wife (hospitality) and it’s also very much her vision. I think it’s easier for him to go along with someone else’s project and complain he’s serving someone else’s idea, than to risk putting his own out there.
And I remember saying to the second friend that he’d caused his wife and children so much pain by ending the marriage out of the blue, that he owed it to everyone to make sure that the life he made for himself afterwards was what he really wanted and chose. Five years on, and the dynamic in his new household is pretty much exactly what it was in his first, and he’s retreated to the kind of sofa-bound, routine-bound life that made him say he was ‘sleepwalking’ through his first marriage.
The marriage wasn’t the issue in either case. Both men just brought themselves with them into a new relationship. I wouldn’t say either is actively unhappy, more defaulted to who they always were.
I think your self-protective instincts kicked in for a reason, though, OP. Listen to them.