You need to play him at his own game and disengage at the same time.
Send him a text on, say, Thursday, to say as per the CO ds will be available for contact on Sat at 10am, will he be seeing him. (Might need to encourage ds to go along with it for 2 months or so). If he doesn't respond, or doesn't show up, text him back confirming that as he cancelled the scheduled visit you will see him in 2 weeks (or whatever). Don't engage with any backwards and forwards crap.
After a couple of months of being messed about you (and ds, more importantly) will have evidence to show that HE isn't sticking to the CO - if ds doesn't want to go message to say that despite your best efforts DS has decided that due to previous x no. of weeks being let down he doesn't want to come. Please let me know when he (ex) would like to see ds in future and I will try to facilitate.
By taking back the control you will begin to feel more empowered, and giving ds a voice and a choice.
Then tell him to take you to court. They won't enforce it if you can evidence that you have rock-solid proof that over a period of x weeks you have encouraged contact but ex has refused.
If you speak to him in person you need to switch to text or email only, send him a message today that you will only communicate by text/email in future (so that you can build up this audit trail).
My ex tried this stunt many times, even refusing permission for me to take the kids on holiday despite only seeing them a few times a year. I took HIM to court for that and he piped down after that. Your ds is 11 and will very soon be able to vote with his feet - doing the above will support him in making those choices whilst still keeping the door open.