oh dear Fuckit, you have (nearly, thankfully) fallen for the oldest trick in the adulterer's handbook "my wife doesn't understand me"
it's top of the list next to "we haven't slept together for years" (yeah, sure), "we live separate lives", "she lives only for the children", "she doesn't pay me enough attention", "I've felt pushed out since the children came along..."
see a pattern here ? A selfish person who is looking for thrills with any willing woman.
his wife is probably completely unaware he is spouting this shit to you
the thing is though, when I say "nearly" you have allowed this man to indulge in an emotional affair with you, say things to you that cross boundaries in a marriage, invest time and energy in OW (you!) that he should have been ploughing into his family
now, you can say that is not your problem...but I think any decent person should feel some guilt about that and not hide behind "well, if it wasn't me, it would be someone else"
the fact is, it was you and you are hiding behind some rather naive rationalisations about why people cheat on their spouses
please wake up, and stay woken up...you haven't ever struck me as the type of woman to contribute to the breakdown of a family, no matter how indirectly