OrionsGirl - I never said you hated your lovely man's ex - but my god you make it sound as though you and your partner did his ex-wife a favour in having an affair, and then him leaving her for you?
Does she see it that way? Did she find out? Was she upset? Or do those tiny insignificant details not matter to you?
And presumably, if he now finds someone he likes better than you and proceeds to have an affair, then leave you for new girl you will be totally OK with that because after all he couldn't been all that happy with you? No? Do you honestly think given his history that you are immune? Are you so different, so special that that could never, ever happen to you?
Of course relationships end and people fall out of love. But the end of a relationship when it is one-sided (i.e. not mutual) is devastating enough without throwing in the grenade of an affair! It makes a terrible situation far, far worse and much more painful and harder to get over than someone just leaving. If you think otherwise you are delusional.
I have a friend whose husband left her and their young child, out of the blue. She was devastated, didn't see it coming at all, he swore there was nobody else, and she was (just about) dealing with it when her husband was spotted canoodling up to his mistress in a pub 20 miles away. I cannot tell you the devastation that wrecked on her, her self confidence and self esteem.
If you are not happy, and want out, then be honest and leave - rather than dipping your wick elsewhere and jumping ship for something 'better'. So many men seem to do it this way - it is like they don't have the balls to leave until they have lined up a replacement. It is a shockingly poor way to treat their spouse.