Well, I'd be tempted to unleash the verbal shit storm that he deserves!
But better to keep your cool.
What an arsehole to drive all that way then deliberately ignore your messages and calls because he takes more pleasure in being a shit to you, than he does in seeing his kids.
As others have said, what you're doing wrong is letting him take the piss.
I would text him calmly and say, "we need fixed arrangements so that there aren't problems like today".
It's tricky because you're the one that has moved quite far away, so generally I would say the onus is on you to do the travel. But I do think it's better for the kids to go to him than him to your new home - easier to control you not being there, and him not being in your house. He absolutely should not be.
I would propose "one weekend a month I will you will bring them to you on Fri at 19:00, and collect at 16:00 on Sun. I will not stay. If you want a second weekend, you can collect them from me at the same times, and take them back to your house. This will start ".
Ignore any demands that you stay. Do not tell the kids on the first weekend what is happening in case he deliberately goes out. Be vague about how long you're going for. Arrange to stay somewhere Fri night so that if he's not there, or refuses to have them without you, you have somewhere to go. Tell the kids you're going to but you haven't decided what you want to do yet. Bring a friend to help you keep your resolve - harder for him to be a doorstep arsehole saying he needs you there with an audience!
He will cope just fine with 3 - he's being an arsehole controlling you playing on your good parenting of your kids.
And get to a bloody solicitor and start your divorce!
I don't doubt he's not paying maintenance currently 
Maybe talk to Women's Aid as I suspect you're dealing with the fallout of an abusive marriage too and may find it hard to say no to him.
Good luck, and good luck with your medical issues. x