Agree with getting some solid visitation in place. Draw up a court order (I did), detailing exact times of drop-off/pick-up. Stick to it ALWAYS. He will muck about with it but, as far as possible, don't let him. Keep him on an extremely short lead.
Dont show any emotion EVER. Practise speaking in statements - v v short. Pretend you're in front of a barrister, don't add one syllable more than absolutely necessary. Less is more.
Don't respond to comms from him unless absolutely essential - if they are excessive, take out a restraining order if he doesn't respond to solicitor's directive to cease. I did. He was also instructed to come no further than my gatepost not that I had one . you need to get him off your doorstep/street permanently.
Don't respond to any apparent softening on his part - he will only be buttering you up to do you over again. Don't think 'at last we are getting to a civilised plateau' - a civilised relationship is not possible with people like this; they always, always manipulate and control.
You have much more clout on your side, his 'clout' is all puff. Convincing puff, mind, but don't be taken in. He's a sad little man, don't forget that (though don't feel sorry for him! Aim to feel nothing.)
You can do this, girl. Have you done the Freedom Programme?
Present to him an entirely unreadable blank slate. Don't insult him - too emotional and indicates something of your personality (I'd Like to say I stuck to this but I didn't always.. But I did MOST of the time)
I relate to the fear. I don't blame you - but don't ACT fearful. Too much emotion. The fear will fade with time, esp if you keep away from him. It takes a while but you get there in the end.
To that end, I would be going for supervised access. Then you will never have to see him. It would be preferable if he had no access to the kids at all - abusers abuse everyone, regardless who they are. You have evidence he is abusing them by abusing you: use that legally.