Maggie - I get where you're coming from exactly. I understand what Labrawoman is saying, but I think you made it very clear in your early posts that you saw this as a symptom, not the root cause of your marriage difficulties.
You're also in a good position to understand how situations like this happen, given your own EA. The problem as I see it is that you are perhaps more self-aware and emotionally intuitive than your H and were therefore able to work through one aspect of your EA (a pretty important one at that) which was that OM was not going to be an alternative and he wasn't worth wrecking your marriage for. You perhaps didn't work through some of the other aspects though - I'll come to those later.
The dissonance in this situation however is that your H, not being as emotionally intuitive (suspect Aussieng is spot on, as usual) might have got himself into a "true love" fantasy. As you say, if he had tumbled you in the early days of your EA, maybe that's where you'd have been too. It depends to an extent on your H's personality, how long this has been going on for and what has really happened between them. I'm a bit sceptical that he wanted to leave before Christmas, when allegedly nothing physical had happened between them - and according to him, still hasn't.
Geekdad is spot on therefore when he says that the presence of the OW, although a symptom, is a major obstacle to you moving on and discussing what are the real issues in the marriage. That situation has got to be resolved properly before you can even begin to do that. We might all be able to see that the OW is an irrelevance almost, but to your DH, she isn't at all. At the moment, he's still idealising her. He can only start being truly objective about your marriage and its future once the bubble of OW is burst. As you know yourself, people in the grip of affairs go through a temporary form of insanity. This leads to them behaving in all sorts of uncharacteristic ways and their decisions and grip on reality are astoundingly poor.
Your H is still in that "insane" stage, which is what I meant in an earlier post about him being in a dangerous place.
Talking to others who are not in the grip of that insanity will help, especially if those individuals are emotionally intelligent themselves. It worries me a bit that the friend he has chosen to speak to does not know you very well - that seems a gap in itself, that a close friend of his, does not have a relationship with you. You on the other hand, have spoken to someone who could be a friend to you both - I think that's a wise (and caring) move of yours.
Although you can see how and why these extra-marital connections occur, I sense your frustration that for years, it was you bringing your grievances into the open and your H who wanted them closed down and concluded. I have a sense that in your mind, he never once raised with you that he had issues - he never told you how he was feeling. He has chosen to express his dissatisfaction by having an affair, which feels unfair to you. Perhaps when you had your EA, one of your justifications was that you had been open about your dissatisfaction, your H had therefore had the chance to rectify matters, had chosen not to (yet again) and therefore to some extent, he had only himself to blame for you seeking affection from elsewhere?
You perhaps feel that because you "dealt" with your EA, recognised it for what it was and "came to your senses", that chapter is now closed. However, the problem with this is that having that sort of secret alter's one's view of your partner. You (perhaps subconsciously) start to see him as a victim - and also perhaps begin a very subtle process of letting him off the hook for behaviours that you previously might have challenged, but now feel too guilty to pull him up about. The dynamic of your relationship changed in some way, because of what you'd done and this secret. Worth thinking about?
You then immediately "atoned" by embracing motherhood - and with all the emotional labour involved in this, there was no time to notice the changed dynamic, perhaps?
One of the things you might also like to think about is that with-holding grievances, refusing to engage in "relationship" conversations and with-holding sex (all things your H seems to have done over the years) are subtle methods of control. You might never have viewed him as particularly controlling, but those behaviours are indeed controlling.
I think what you intend to say to him is right - that you will fight for your marriage, but that there will be certain conditions. You might perhaps also like to think about the viability of them still being in contact at all - I honestly don't think it works, Maggie, especially at this stage of their affair. It's a bit like an addiction to him at the moment and the only really effective solution is cold turkey. Many women have reported on here that the true rebuilding only started when there was no contact whatsoever with the affair partner.