Who declared their feelings first?
You see, in the situation you describe, it is rarely if ever the case that two marriages will be as bad as eachother's, or that feelings are matched. It's likely that your feelings are stronger, his relationship isn't as unhappy as yours (if it is at all) and so he can switch off more easily.
What you've probably experienced is something called 'mirroring' where he is too polite and kind (to you) to be honest and say why he prefers his wife and their relationship to you. That's an unkindness to you because you don't have closure and are left with a fantasy that he would be with you if he wasn't such a 'decent man'.
And he isn't being kind to his wife at all creating this illusion, in order to stop him looking the bad guy to you.
Better to conclude that if he wanted you enough, he would be with you and that there's far more feeling for his wife than he's been letting on - and fewer feelings for you.
BTW, men and women are no different at all. If there'd been enough stacking up in your favour, he would leave to be with you, just as you would have to be with him. People are rarely unselfish enough to stay in a relationship if there's nothing in it for them, particularly those who feel entitled to an emotional affair.