I have rejoined specifically to reply to this. Your ex is an arse, I had exactly the same scenario, my kids were 11 & 12, they are in their 20's now. He had a girlfriend that was just awful to them, they didn't feel safe, my youngest didn't want to go as he was worried "something" might happen! He was wound up before every visit, I got them a phone to use as my ex refused to allow them to use his landline, I got vague threats about him stopping them using it, they texted me when they were unhappy, i.e girlfriend calling them little shits, calling me a fucking cow, their dad shouting at them, threatening them with smacking, or if there was a bad atmosphere as the girlfriend was whispering or giving them dirty looks, I encouraged them to go, I tried my best to facilitate a relationship, you know what, I was wrong, he hated me more than he loved them, he was with someone that saw them as interlopers in her life, I eventually kept a diary of everything that was said, they wrote in it for every visit, in one year there was one visit that went without any incident.
I wrote a letter to him, I explained the children were anxious, I offered mediation for him and the kids, with me coming along as an option, to allow them to discuss why they felt anxious, I offered him day only visits as overnight stays were too stressful, I explained why his girlfriends actions were unacceptable. Nothing worked, the visits tailed off, once he realised they could not be persuaded I was a bitch from hell he wrote them off.
More than ten years later he has married the girlfriend, my children, now adults, hate him, and have nothing to do with him or her. His loss. My only regret is I tried to keep them visiting even when they told me they were unhappy.
Tell him you will ask for a referral for mediation, put in writing that you are willing to work this out, but your children need to feel safe and secure, and that their wellbeing comes first, if they need to phone him when they are with you there is no issue and you expect it to be the same for him, your children are not possessions and if they need their parents they should be able to speak to them. By the sound of it he is going to bow out of their lives, but blame you, your children will realise this, and my advice is to keep diaries. I brought out my 2003 diary the other day, I was going to throw it out, my eldest read it, said they had forgotten a lot of what had happened and asked to keep it, said they thought it was an important record of their childhood.
And good luck, I hope your children come through this relatively unscathed.