Oh and he gave a sly dig that when i get home from work after a 9 hour shift, i sit down and apparently that makes me lazy.
Ah yes, that's well known, it's called DARVO - Deny, Attack, Reversing Victim and Offender.
Basically accusing you of doing what he's doing, to confuse and unsettle you.
Common behaviour, often described on Mumsnet.
And the bit about what are you doing with your money - well, what's he doing with his? Bit more DARVO there, perhaps?
I suggested we part ways and he got so nasty to me not abusive, but starts saying I couldn't cope without him and he'd phone social services on me.
And that's a top tactic of the abusive coercive controller.
You taking yourself and the children away from him = depriving him of property he controls.
Too many men believe women and children are things. Things they own, and can do what they like with and to them.
The thought of you all not being there is in the same place in his mind as the thought of his car or his phone being taken away. It's about fear of losing property. Not relationships, not love.
So he issues threats designed to prod your worst fear - losing your children. Social services = danger, for you. So that's where he goes.
To control you.
And telling you you can't cope alone? Classic examples of belittling and undermining you, brainwashing you to try to make you feel / believe you genuinely couldn't cope alone. That's called gaslighting - yet another tick on the controller's to-do list.
And yet coping is precisely what you ARE doing, day to day. Raising your kids well, without his practical input and help.
Worried because you need the money he does make? As their father he'd be legally required to contribute, whether you live together or not.
And the threat to bang around and wake you all up at the weekend - does he honestly think he'd be able to magically get out of his slobbish habits each weekend specifically to upset his family?
What a prize shit you've got there.
So start thinking who DO you have? Your parents? Siblings? Friends? Any help available though work?
Get a solicitor who can tell you your rights. Do not listen to ANYTHING he tells you about your rights, his responsibilities, (or vice versa), your character, your capabilities, your prospects etc from now on. He is .... an unreliable source of information.
All the best. We're here and translate what he says and does.