I don't think it's simple at all, nothing is. Life isn't simple for people who don't have affairs either.
But the reality is that using that as an excuse to continue toxic and unhealthy behaviour does nothing but perpetuate that behaviour.
The 'nothing is black and white' argument falls down because most people understand that nothing is black and white, but many of us still make decisions that aren't damaging to ourselves or others despite all the grey areas we all have to navigate.
He is not a good man. You won't have won a prize if he ends up with you. He will only be with you if his wife kicks him out and he doesn't want to be alone. You're being complicit and selfish currently. You will feel so much better about yourself if you take the active choice to end this rather than waiting for him to do so and pining after him.
He's with her today, maybe looking at baby scans and talking about names and how they'll decorate the twins' room.
I'm not saying that to be a dick, I'm saying that because that's his real life. You are his secret life, the life he escapes to because reality is hard for everyone, but he isn't brave enough to leave for you or doesn't want you enough to do so.
I know that's shit to hear but dozens of people telling you yes he does adore you and it's ok to wish he'd leave his pregnant wife wouldn't do you any good.
You need some tough love from people who care about you and want more for you than this, but I'm guessing nobody knows?