Hi Emily. It sounds like you're doing the right thing, putting your dc first and also ensuring your physical safety. I agree with Nats about recording your worries and fears with as many official parties as possible. If your dc is 5 you could speak to the school just to give them the heads up as regards your situation. I think the Health Visitor is also a good person to speak to.
As is said so often on here, you only have to facilitate contact between your ex and the dc. Your relationship with him is over, you don't have to see or speak to him. You don't have to tolerate being sworn at and called vile names. Whenever I see a man has called the mother of his children the "c" word I always think he has no boundaries.
I think you have good reason to withhold contact for your youngest. If they're bf and have health issues, how is he going to be with them for any length of time without you? Unless he is prepared to treat you with respect and meet in a neutral place (ie not your home) I don't see how you can arrange contact. He's the one putting up barriers through his behaviour, not you. Of course that's not how he's going to frame it to his family. But you know the truth, and so does he, which is what matters.
My advice is to go in hard from the outset. If you don't, he sounds like the type of person who will try and manipulate and threaten you from now on. The more he gets away with it the more confident he'll feel to continue. Are you in a position to see a solicitor? A simple letter stating that you are not withholding contact, and then setting out what contact is possible at the present time. You can say why you are not prepared to be alone with him (physical aggression, and then quote some of his texts). You can also say he is only to text you about information relating to the dc. Anything else you will ignore.
If he then says I'll take you to court, do you feel you could call his bluff and say fine, do so? Does he have the finances, the energy, the wherewithal? The stuff about the custody is rubbish as you rightly say.
Nats Are you going to be on the red carpet at the premiere? The Daily Fail might do a piece about your dress ! Seriously, that sounds good, hopefully it will attract some publicity. Do you have another date to see Piers Morgan on the GMB sofa?