I said downthread what he was angry about here:
His anger is less to do with being neglected and more about being found out actually. He is literally furious with you for rumbling him and the more indignant he feels about that and having to accept he was in the wrong here, the more he is going to counter-attack with "reasons" for why he behaved as he did.
I'm glad the sheer weight of common sense in posts this afternoon has made you see sense and I hope now you will stop blaming yourself. And look, I'm sorry, but you've only got a proven liar's word that he hasn't been unfaithful before. What that might actually mean is that he hasn't had the opportunity before (i.e. found a willing partner) or that he has indeed been unfaithful but won't tell you about it.
Trust me on this. There are plenty of people who have affairs despite having regular sex and happy marriages. The one thing an existing partner can never be though is new and this is frequently all it is, when it comes down to it.
Sufficient has also bravely shared that despite her putting her all into her marriage and trying to be every thing her husband complained that she wasn't, it still didn't matter. I often point out that you could have been the perfect woman, performing sexual gymnastics every night and your H would still have been unfaithful, because honestly - none of us can prevent someone else being unfaithful.
If you don't believe me, please do ditch that Andrew Marshall book and read Shirley Glass instead. In her much longer career as a therapist and her far more rigorous research, she concluded that the prevention myth is one of the most damaging ones of all. There is simply no evidence to support that either a partner or a happy relationship can prevent infidelity. There was nothing you could have done. Your H was probably just as likely to have been unfaithful to you before the DCs came along, had he had the opportunity and the means of evading detection.
This is because he has an individual vulnerability to infidelity and this time, he had the lifestyle to support it. Lots of trips away from home overnight, a social life without his wife, a trusting partner who would never invade his privacy and much as it pains me to admit it, I suspect he did this to you now because he knows you are in a more vulnerable position being a SAHM with three young DCs and dependent on him. If you'd had financial independence and no DCs, he might not have risked messing you around as much. He knows that men are often forgiven for affairs "for the sake of the children" whereas many women would walk if they had the wherewithal and the independence.
Keep reversing this all the time. You weren't getting enough from him, but you didn't have an affair. If you had come on here complaining that your H was neglecting you and so you'd started an affair, I would be saying the same to you about personal responsibility - and have, many times to delusional unfaithful women looking for an excuse for their own behaviour.