Canigetanythingright this t**t is the cause of you being at breaking point. I have no idea what you need to do to recognise this, but you need to stop caring about him. He has you dangling on a string (causing your distress)
Take control. Write down terms and conditions and most important, stick to them. Tell him the world has changed. Its important to tell him things have changed.
As an aside - I am not allowed to know where he lives personally no one (including the other parent) would wonder off with my kids if I didn't know where they lived (but thats up to you).
feels like a Wild goose chase of course it does..so stop chasing.
Arrange a pick-up time and write it down (tell him he has 10 miuntes grace to be late), after that you will leave and your phone will be off. Then if he is late, leave and turn your phone off. End of, no more discussion.
Also if he misses one meet, he has to email you and arrange next meet.
Start to lay solid ground rules and stick to them. Stop him playing games and having the upper hand.
As I said above, its sad but you cannot create a good Father, no matter how much you acquiece.