You likely split up because he is an abusive git and he's still abusing you. What was the point in splitting up? To minimise the abuse to 2/7 days instead of 7/7?
You need to make big changes.
He will HATE them. But after a while, if you're consistent, he will get over it.
Email/text him. From now on, document everything.
Tell him that he will have contact every other weekend as this is what is best for the kids (from what you've said). If he has an issue with this, he can be the one to pay to get it settled in court.
Tell him that his contact will no longer be taking place in your home. He can either arrange to take the kids to somewhere like soft play or a familiar family member/friend's house, or contact just isn't happening.
Meet him at a neutral place for handovers. Don't do handovers at your home.
Arrange for him to call the kids every evening if he and the kids want this.
Is there any reason why he can't rent out a decent home where his kids are actually allowed?
Do you trust them with him unsupervised? Sounds like he really upsets them tbh.