Hi Spook. What a mess. There may be things your dh has not been happy with, and it might be quite valid to mention these - but his reaction to it is his responisibility and no one else's.
If someone has grievances with their partner, they should deal with it inside the relationship. If my dh was unhappy with something in our marriage I would expect to be given the chance to make my own mind up whether or not I was prepared to do something to change things. It is not fair of him to resort to an affair and then tell you it is your fault.
If only he could be made to understand how devastatingly hurtful it is to be betrayed and lied to. He is still covering up his guilt by looking elsewhere for an explanation all the time. If I felt so upset with my dh that I wanted to start another relationship I would break up with him first and then start with someone else - no matter what he might have done to cause my unhappiness. I am responsible for my own behaviour.
Your dh is trying to distract you and himself from that very basic principle. I wonder if it has occurred to him that a man who is conducting an extra-marital affair is bound not to be giving the attention and care to the marriage that he should. I would be impossible not to steal from the wife to give to the mistress in terms of attention, time, effort, energy, patience, affection, etc. If, therefore, he has not been putting 100% into your marriage, he can hardly complain that you haven't.
I think it is worth listening to his grievances and discussing them, but no way would I accept the blame for the affair on that basis. The responsibility for that rests with him and him alone, and until he is prepared to accept that and act accordingly, you are going to get nowhere with trying to rebuild things.