Not relevant directly to the OP but yes an affair is kind of more intense, not because it's 'naughty' (I never understood that) but because you're both feeling as though you're somehow being torn, you're unhappy, and when you're 'suffering' something together that is a very powerful bonding instrument.
It's because it involves pain. Having conflicting feelings, wanting very much to be with someone but being 'denied' that because of your prior loyalties is a painful experience, and to see your married lover in pain through this makes you want to take it away, to make it better, to soothe them.
And you will have your own pain, from being unable to access them in the way that you desire.
The shared suffering is something that can draw you together even more strongly. What it conveniently sidesteps is the fact that each of you has an actual choice, about whether you are together or not. Believing a man stays with his family due to loyalty alone is a big mistake. As is being with a man who sees his family as his prison wardens. It's a victim mentality. It often goes alongside other things such as alcoholism, anything that he can blame some other factor for, rather than taking it onto himself.
He probably does not realise he is doing this. and it is easier to pretendthat he is right, and he has no choice and needs you to take away his pain once in a while.
It doesn't mean you aren't in love with him; it just means he loves himself more than he loves you.