It's relevant because either you were taking time he should have been spending with his family for the affair (and I'm not convinced the 2 of you didn't, I'd love to hear ex wife's side of this sordid little tale), or you were taking time from your enployer, which is unprofessional, effectively theft and unfair on other colleagues too.
I'm also guessing when you say the ex wife "did rather well out of the divorce" I'm guessing you mean financially. Money isn't everything and certainly doesn't make up for the hurt and betrayal caused by your affair. No amount of money can. Plus it's actually rare for the ex wife to be financially better off after a divorce and we only have your word for it. Again I'd love to know her side of the story.
Re children and the effect on them. I have a 17 year old who 15 years after the split from her father is still very much dealing with the ramifications, and I've seen this be the case for many children of divorce. She doesn't trust boyfriends, still feels hurt by her father not thinking SHE was important enough to him for him NOT to cheat, and we've certainly had to deal with the financial effects too. While he and his 2nd wife, enjoy a VERY comfortable lifestyle which frankly if I hadn't kicked his butt into putting more effort into his career, planning for the future (investments etc) they would not be benefiting from - while he has never paid child maintenance regularly or in full, hasn't seen his daughter for several years through his own choice, I believe because it makes him feel guilty.
Even in families where the children do still have contact etc there are still major trust issues and a lot of hurt. That NEVER goes.
I wonder how old his daughter is and what hard conversations are going to have to be had in the future.
And frankly I hope it makes you both feel the guilt of your actions as you deserve to.