It has been put to me by my GP, a divorce lawyer I was consulting to check my options if I divorced (live abroad so complicated) and two separate psychologists that I could have an affair. All know the exact details of my marriage, which I won't go into online, but simplistically is that I am married to a very lovely man who is asexual (who it turned out pretty much faked sexual interest before marriage) and for a number of complex reasons cannot get divorced right now (think visa requirements etc).
I have asked to be allowed to take a lover, he won't accept. I cannot betray him - I love him. A lover and husband is also not my dream relationship setup.
But sometimes I wish I could have an affair. To be held, to not feel completely undesirable and physically rejected and useless. To feel a body next to mine that wants me and mine.
But then I think that aside from my personal reasons not to, everybody would judge me when someone finds out, and like the OP who distances herself from cheaters, I'd be ostracized. That makes me very sad, because by the time i am able to divorce, I'll have barely had sex for my entire 30s, but nobody knows.
So don't judge cheaters. Sometimes, not always of course, there may be a very good reason, and the "cheated on" partner is unlikely to ever tell you what it is.