Oh baby!
How to minimise, patronise, infantilise, and basically annoy the fuck out of me... not a good start.
Hi Wait. You are not the problem. You are not stupid. You are not mad. You are not the one throwing away the marriage, he is. It's him! He is the one who is blithely ripping up the marriage, whilst blaming it on you. What is he doing to resolve the problems? As you say, has he been to the GP yet? He's the one who ignores you, witholds affection, fucking texts you from downstairs...
You are right, you have had enough of talking, followed by shouting and then being called mad. Sounds spot-on to me. Nobody would fancy that.
He is the one being so obnoxious that you feel absolutely no desire to interact with him. Frankly, no sane person would. He needs to shape up, because it is his behaviour that is unacceptable.
Have you called him mad, have you shouted at him and bullied him in front of the children? Nah, thought not.
Stay calm, disengage and detach and make your secret plans. Just don't let him mess with your head. When you get that feeling, tell yourself consciously 'he is just trying to mess with my head', register it and then go about your business. It helps.