Making you feel like you're going insane is his intent.
Some people feed off other people's distress. They are parasites, predators, vampires.
And they therefore stir situations that trigger your distress - and deliver their drug.
This is now the only way he has to get at you, now you've escaped him. Using the children. So he's focussing all attention in that.
So you need to ensure you keep all the evidence.
Everything in writing - no phone calls. If the children aren't with him, leave his calls to voice mail. Keep and file all voice mails, texts, emails, letters, messages. Log all interaction with him. Consider your 'relationship' with him as a case, and keep case notes.
Have you done your best to facilitate time with the children? If you can prove it, great, if not, start making and creating evidence. Be careful what you say about him. Keep it factual but appropriate. Log what you tell them.
Set up a schedule for him to visit, meet somewhere neutral, whatever suits best.
If he doesn't turn up, tries to change it, complains - keep the evidence.
Log everything that happens.
Keep your solicitor aware.
At their ages, it's only a very short time before they decide entirely what they do.
He's collecting flying monkeys - people he's winding up to do his dirty work. It's none of their business. Politely shut down any attempt from them to interfere.