Well. It's a good start to recognise he is NOT good for your DC. That's a start. None of this 'kids need father's/I must hang on a cross for their sake'.
Yy kids need fathers - but they don't need abusive cunts who use the children to abuse their mothers. That's not a father.
Is he the biological father? Were you married? Is he on the birth certificate? Talk to Women's Aid, get them onside. Have you done the Freedom Programme? Do it NOW if not.
My turning point came at a WA DV support group (now defunct 
) and, although we all shook with fear and trauma so the coffee table was awash with spilt drinks we shook so much, we ALWAYS ended up laughing and laughing, crying with laughter and derision at our fuckhead,stupid, absurd, thick abusers. Every week we'd arrive in a mess, crying - and every week we'd end up crying with laughter.
Bcs abusers are so stupid. So nonsensical. They're little tinpot generals all puffed up with ridiculousness.
I'm trying to remember something we laughed at and the only thing I can remember is the time he took DC out for school shoes and, I kid you not, he must have texted me 20+ times with endless stupid questions. Sounds small eh. We laughed at his idiocy, his blatant attempts to beleaguer me and wear me down. (If anyone is saying 'but he bought the kids school shoes! Couldn't have been that bad a father!' - he got the shoes bcs at that stage he wouldn't let me have any money to buy them myself. He was extremely rich and he starved me of cash, I was extremely poor.)
So I suppose the turning point for me was contempt, my contempt for him. I saw he was a measly little man, a stump of a thing.
So, back to contact: why is he seeing DC every weekend? Who has decided that? Is there a legal schedule in place? Or does he call all the shots. Stop that if so.
Ultimately, as he is poison to your DC (in all ways, not just some ways), drag your feet over every area of access. Don't be compliant 'for the sake of the children' bcs that is misguided: it is not for the good of the children but for the harm of the children, so don't do it. If you are worried about money, him not giving you maintenance (to punish you) then face that and live another way financially. Of course he should pay maintenance but at what cost? The cost is too high. Better to cut off that avenue of abuse by finding another way to get by. Frankly, a shed is better than the immense toll abusers put on one, the extremely high cost to ourselves and our children.
Cut off every avenue of potential abuse. Find a way. Remember, he's a grotesque little thing, no use to man or beast.