Hmm. From what you've put in your posts OP, I'm fairly certain I would be able to point out where your ex lives. I am 100% certain I don't know him, but he would be fairly easily identified from the information you've given. I really hope you've changed some of the details, I hate to think his wife or similar would recognise him, and it would become extra shit for you.
His wife and he are possibly playing the long game here. Move further away, refuse to help make contact visits possible. You will at somepoint say no, I can't do this any more. I can't afford/other reason.
He will use this as an excuse to get you to court for being in breach of the contact arrangements. From my understanding they can at this point have social services involved for the emotional distress caused to the children (I can't remember it 100% but I was threatened with it when going through similar to DDs biological father)
When you say, I was happy to. I just couldn't afford.... when he would exclaim, but I give you money for that. How can you not afford x,y,z?? What are you doing with MY money?
The argument would then go, if she can't afford to make kids available for contact, maybe they're better off with me? All with a fancy lawyer.
You won't win this one. I was dragged through court again and again by DDs dad who didn't have a pot to piss in.
Just hold it together. Don't tell him you haven't got. Just make it happen, somehow. Don't give him any chance to consider you vulnerable.
PS- if you ever need to move out of the city Bedford is still fairly cheap. You'd not need to worry about travel cost either. X