If anyone is being selfish here it is your so called H. And he is abusive towards you (and in turn these children) to boot. Of course he does not want you to leave, he merely wants you to continue to skivvy around after him like you have been doing. You were vulnerable to such attentions from this other man as well partly because of your childhood but also because of your abusive, not to just say crap, sounding marriage now. Is this really what you want to teach your girls about relationships, no it is not. They as well as you deserve better.
Re your comment:-
"I struggle with self esteem and things from my childhood I don’t want my girls to be the same".
What did you learn about relationships when you were growing up? Sadly no-one saw it fit to protect you from life at home and it may be an idea also for you to talk to NAPAC (link here) napac.org.uk/. What happened to you was not your fault.
Of course we all want better for our own children but your actions speak louder than words here. Who or what are you staying for now?. What are you getting out of this relationship?. What do you want to teach your children about relationships and what are they learning here?.
Show your kids properly that the only acceptable level of abuse in a relationship is none. For your part you're showing them that currently at least this from him is still acceptable to you. He's also having a pop at your eldest daughter for being clumsy, he is a truly nasty bully of an individual and such types too hate women, ALL of them.
Re your comment:-
"Anyway he was threatening to kill himself if I didn’t come back (I was only away 1night) and how much he loves and he will help around the house blah blah! So I came back and for a week he did help but gradually it’s going back to old ways. I feel down about it all and I worry my life will be like this forever".
It will be like this if you do not take steps yourself to break free of your abuser. How can you be helped into doing that?. What do you need?. Abusive men often threaten to kill themselves; they say this because they can and it works for them. He is a master of manipulation and you were truly manipulated here. What you should have done is say fine then pick the phone up and dial 999 to get an ambulance then leave him to explain himself. That manipulation tactic of his worked here because that also brought you back to heel. He just gave you any old flannel to shut you up; he has no intentions of changing his ways and become a decent sort of man or father to his kids.
Re leaving, feel the fear and do it anyway. Put a plan in place with the help of the likes of Womens Aid and the Rights of Women organisations to leave your abuser. You write that he works away so use that time he is away to properly plan your escape and divorce from this non marriage you write of. If you can go to Boots many of their consultation rooms have information and access to domestic violence organisations also.