Your H was definitely in the wrong by seeking emotional/physical support in another woman, but it does seem like your marriage had some serious issues which would need working on regardless of whether he did or didn't.
You point out that you had 4DC because you wanted them, even though your H didn't. And presumably the extra mouth(s) to feed mean that he feels committed to his long commute and job when perhaps he would have preferred to not be the main breadwinner and not have more DC so you could work as well and share that burden.
You were unhappy about the home "drudgery" but perhaps your H felt that was what you wanted as you were so determined to have 4DC. Presumably after no. 3 you weren't saying "I hate being at home, I'm knackered all the time, I'm bored of no adult company and I want to be working rather than a SAHM"?
You also say that you rejected his cuddles and sexual advances, which is fair enough as no one should have intimacy they don't want.
However, I think if a female OP had said "I feel trapped in my job, my H wanted 4DC when I really didn't and now he's at home with them everyday and I'm tied to a 3Hr commute and a job I don't like, then when I get home he won't even cuddle me, I'm so unhappy and I tried to speak to my H about it but he doesn't listen, just says how tired and fed up he is, when this is what he wanted..." there would have been a lot of sympathy for that woman.
As I said at the start the EA/affair was definitely wrong and inexcusable, but I think your whole relationship has some grass root problems that need to be resolved.