You've chosen to spend your life with a man who:
Lied to and cheated on his wife, so you know he is perfectly capable of doing it to you - because he has developed a particular set of skills, plus the habit of having a bit on the side
Is making no effort to move on and accept that this is in the past, and you suspect he regrets what he did and what he threw away
Keeps scratching at the scab - and you believe it's because he can't let go and is trying to find a way back to her
Has an ex who despite all this keeps going back to associate with the man who did this to her - and you believe that it's because despite that she can't let go either, and is trying to find a way back to him
And is failing / refusing to empathise with and respect how you feel as his actual partner.
There are several terms for what you are in this, OP.
Piggy in the middle. The third wheel. The gooseberry.
I'd bet if you ducked out, they'd be back together to try again.
Is it worth it? Really?