I suspect that your husband had a bit of an inferiority complex going on when the marriage started - liked the way you could suppliment him but probably, deep down, had some resentment there - the references back to not trusting you feeds into this.
You were doing a great thing but alot of sad sack men like your husband take a wifes success as some kind of reference to their own masculinity, rather than celebrating it as a wonderful achievement and a sign that they chose a wonderful partner.
It shows that, even then, he was really only capable of seeing this through how they made HIM feel.
So when his business took off it probably pumped his little ego. Made him feel the big man, made him the pompous selfish self congratulatory Big Man he knew he was and, as his income overtook yours he was able to look at you in your now 'little job' with some contempt - he was the winner, he was the Best. And that overinflated opinion of himself was exactly the kind of thinking that would allow a man like that to have an affair.
Im sure your marriage was OK. Nothing worse or better than anyone elses. It also served a good perpose of facilitating him, and putting a good face out to the public. But once his head was turned things would change as DMBL indicated. Something new and shiny presented itself and everything else dulled in comparison of a scenario that was always going to be hormone and adrenaline filled.
He thinks he is amazing. He thinks he is the Big Catch - that much shines through. He is absolutely floored that the romance of the century has had the plugged pulled and it is humiliating and unbelievable and has him snarling and self pitying and holding out hope. I dont believe for one minute that he is sitting sad and lonely and crying in his room. Thats pure emotional blackmail.
As far as he is concerned the plan was that he would break up your marriage, which of course would be devastating for you, because how could it not be, to lose such a man as him - in his mind this was already done deal, but your pain was worth it for his freedoma nd new life.
However things have not gone to plan, and now he wants you to be grateful that he is willing to come back. Why he is cross and angry is because in his mind the fact that it happened and should be swept under the carpet is small potatoes compared to what he has saved you by not leaving. Because his mind has already dealt with your pain of him leaving, but now he wont. You should be happy. You should be pleased. Because as an abuser/narcissist you just dont have any emotions or feelings beyond the ones he ascribes you in the Film Of His Life. The way he is rewriting the past and then storming out of therapy when you challenge it, or keep bringing the OW up, just demonstrates that he is ONLY interested about his feelings here. He cannot hear yours. He doesnt care about yours. He just wants to roll it all back and for you to act as if it never happened.