Hi Sheila .... not sure I can give you any really constructive advice but your ex sounds rather like mine. My son is now 15 but we split when he was 4.5 .... we weren't married either.
I can't accuse him of totally disappearing like some men do but for a long time, I really felt it was a case of "out of sight, out of mind" for my ex. As a result, most contact was at my behest and I sometimes felt that if I didn't nag as much as I did, he'd have completely forgotten about our son. On top of that, for the last 10+ years, he's never made much of an effort re: calling our son during the week just to let him know he's thinking about him and, he's moved 3 times since the split, getting further and further away each time, meaning that contact becomes more practically difficult and he certainly wasn't able to "babysit" for me because of the distance involved. I used to feel very resentful because he earns a very enviable wage (far far above average) and therefore affordability wasn't the reason for his moving away. On the face of it we had "agreed" contact should be every other weekend - but it often felt like only one of us had "agreed" since my ex used to cancel/alter/cut short this contact time whenever he felt like it - irrespective of any plans I may have made for myself !
However - though he probably even now considers me a nagging old bag, I never let the subject drop .... aside from phone calls, I've written numerous emails & letters over the years, concentrating not (much anyway) on how his attitude affected me but emphasising how much his son wanted to see him and how he was disappointed when let down and so on. I was always assertive but never insulting, and took care to write very eloquently, politely and without direct criticism, because I didn't want to give him any opportunity to criticise me ....
.... when someone "confronts" you - albeit "nicely", it's quite difficult to wriggle out of and if he ever said he couldn't see our son this weekend, I'd try to make sure he wouldn't be let off the hook by getting him to commit to the following weekend, all the while emphasising "you are sure aren't you, as I have plans then (even if I didn't)".
Little by little, over the years, this polite nagging has paid dividends. Okay - so I shouldn't have to do this at all, but for the sake of my son, it's not really such a hardship and I admit I like seeing my ex squirm when he realises I've got the measure of him. Don't get me wrong, I don't think he's a heartless dad and he does love our son, but let's just say if someone else - i.e. me - absolved him of his responsibilities by saying it wasn't a problem if he couldn't see our son for 4 weeks, he wouldn't make much of an effort to rectify things off his own back.
Now my son is older, it does at last feel like I can relax a bit and communication between the pair of them directly is improving. Maybe he finds it easier to get on with teenagers than younger children .... not that that excuses less interest when he was youn ger of course.
There is nothing legal you can do to enforce this ..... but, providing your ex isn't a complete uncaring you know what, you can try appealing to his better nature and take control of the situation by asking him to confirm w/end arrangements within a timeframe suggested by you. If he doesn't do this you are then free to make your own arrangements and if it doesn't suit him - tough ! Obviously - I appreciate you want time for yourself too so don't let up on the (polite) nagging .... tell him often enough how dd misses him, wishes she could see him more, talks about him loads, how she'd like him to take her (insert activity here) at 9.30am Saturday (to force him to get up earlier) etc., etc., etc., ...... yeah, I know this is tantamount to emotional blackmail but all in a good cause I think. Don't be nasty to him yourself (or he may dig his heels in) & flatter his ego a bit re: what a great dad he is and you may find, like I did, that he will start to turn.