You know what rotool, I was going to say something about how would a woman feel if her DH has a mate that had had an affair, and I have to say it'd make me 
But then of course my P would just tell me not to be so silly and would go anyway.
My P takes exception to some of my friends... well all of them actually, but I just ignore him cos if I lived the way he wanted me to, it'd be no life at all.
Sadly rotool, it sounds like you ARE living the way he wants you to, and it's causing you immense stress and unhappiness.
And this is before we talk about the effects it will have on your children. You are teaching them that the way he treats you is OK, when clearly it is not. Even your Dr says so,and as AF says, there is no way on earth a Dr would say that without bloody good reason.
I'm not denying that your DC don't adore their father, but they will still love him even if you don't live with him any more.
If you continue to live with him and they cotton on to how controlling and manipulative he is, how unhappy he makes you and how powerless you are against him,(and they WILL!) that love for him may be seriously compromised and will set them up for problems and issues in their own relationships. Your DH is teaching them how to behave and how to be treated.
Darling I know it's hard, we all know it's hard, but you have to do what is right for you, for your DC and for your sanity.
FGS, don't let him cut off your life lines. They are called that for a reason, especially in cases such as yours, he is slowly choking your life.
Please get help, please get out.