In my family, when I was growing up, there were no affairs or divorces except Poor Aunt Mary. My dad would speak of her in hushed tones and was visibly uncomfortable: she left a marriage in her 20s after all of two years, and then had a baby out of wedlock with the man she then married and has spent the rest of her life with. Shocking, I'm sure you'll agree.
(I know this isn't an affair: I am building context.)
We found out when I was 22 that my uncle (on my mum's side), apparently happily married with two grown children, had been having an affair for the best part of 30 years. It'd been a running joke that poor Uncle J worked so hard he'd fall asleep on the train and miss his stop, and have to come back really late; when my granny was dying his wife was furious that he was doing so much and my mum still wasn't grateful (obv. he was with his mistress and lying).
Uncle J also eventually married his affair partner; they have been together ever since.
That's it. So far as I know! 
I slightly wish there were more. My dad (who, btw, is 76 so not actually Victorian, though you wouldn't know it) considers me the black sheep amongst his children, because I left a marriage in my 20s (no kids; amicable; no affairs, goodness you'd think I'd murdered someone), and then split up with my child's other parent in my 30s.
There are, I do believe, worse things than affairs (not that I am condoning them).