If you were my daughter I would not rest until you and your child were away from this man.
What is your relationship like with your mother OP? Have you discussed any of this with her? (I understand that the partner swapping incident may not be something that you want to share with her) But generally, your marriage and the way your husband treats you. Do you have any close friends you have or could confide in?
and he is a great father
What does exactly do you mean by this?
I've noticed that 99.9% of women posting here say this about their partners - even if the men are vile and emotionally abusive or general wastrels. Seems to me that a lot of women mean by this "he obviously adores these small people he has created and enjoys playing with them". That is not a great father; that's just someone who enjoys an ego trip of the worship of little children who are his.
A great father is someone who puts their childrens interests above their own; who will muck in with all the day-to-day boring functionality of running a family (nappy changing/feeding/taking child to football/ballet lessons/dealing with middle of the night incidents etc); ensures the children have a stable family environment (where were your children during the partner swapping incident btw???) but most of all is a good role model in they way he treats their mother with kindness and respect.
I think Ive given the impression that my DH is awful and abusive but he isn't. We rarely argue, we laugh a lot, DH was amazing during my PND after our child was born
I think you are confusing issues here.
Abusive men are frequently charming, particularly publicly, it's how they get away with it. Think about it, if he'd say to you "you're too fat to fuck" on your first date, would there have been a second one? I don't think so.
Laughing a lot is trivial -if you want a laugh, watch some comedy. Feeling utterly sh*t about yourself on a fundamental level because of what he has said to you and how he has behaved towards you is not trivial. It affects the core of your being.
The fact that you still think he is a good husband speaks volumes about your own self-esteem and (probably) his control over you. Let's face it - he's doing pretty well to force a four way partner swap when his own wife was seriously objecting - that's pretty good control I'm afraid to say.