FWIW Mrs Prism and I are together because we had an affair. Well, I had an affair- she wasn't with anyone else. I have been through a lot of opprobrium, some of it imagined, some of it real.
It would be nice if people who are dead miserable in their relationships could simply say "I think it would be better if we parted" and got on with that, before getting together with someone else; but in the real world that is not always, or even often, how it happens. I wish I'd had the courage to do speak my mind, rather than have an affair, but the end result would have been the same.
My ex and I now get on pretty well, having had a number of the conversations we should have had a long time ago. Neither of us wishes we were still together, but we managed to be parents to our DD without rancour.
For the record, my ex has just divorced her third partner (and second husband) since we parted, while Mrs Prism and I are still together (12 years later). She is a very good step-parent, as I think I have alluded to here on MN before now. I've become a better parent because of having to take her views into account (which are sometimes better than mine).
I know one or two really good parents and step-parents who got together though infidelity, and this idea that the way you got together taints you as an individual or as a couple for ever, is rubbish, in my opinion. If you are happy together, there is no reason you can't be just like any other well-suited couple, and woe betide anyone who suggests otherwise. Judge not, lest ye be judged.