You don't need to go to court. Your dc are being abused, you should stop contact and offer a contact centre instead. Otherwise you could be in trouble for knowingly putting them at risk.
Let your ex take you back to court if he is unhappy about that and tell the court why then. In the mean time write to him, via a solicitor or ask your Citizen's Advice Bureau to help you write the letter. Send your letter recorded delivery and keep a copy.
Keep your letter factual and to the point. Explain that you are stopping contact because you believe he is abusing the dc and you do not regard the current contact to be in their best interests. Offer contact at a contact centre in the mean time and suggest mediation as a way of discussing the situation further.
Doing that will show you are trying to be reasonable if he takes it back to court.
Document everything, times, dates, location, what happened. Keep texts and emails and anything that you may need in court should it come to that.
Contact is for the benefit of your dc. No court is going to put you in jail, least of all for trying to protect your dc. But you need the evidence to support your reasons for stopping contact and offering a contact centre and mediation instead. Make sure you have it.