I split from my EXDP almost 2 years ago now, he was exactly the same as yours for the best part of the first year or so, and yes, he was also controlling (or tried to be) during the relationship.
Like yours, he wanted the flexibility of being able to drop in and see the DC whenever he felt like it, so refused to agree anything initially.
I wanted him to do every other weekend, plus half the holidays.
So far, I've managed to get him to agree to every other weekend (so 4 regular days a month). I did it by only making the DC available at those times. If I hadn't heard from him before the week in question started, I cracked on with the plans we had.
If specific events were happening on those weekends, I also confirmed those as well.
Inevitably EXDP would ring on a Wednesday wanting to have the DC Friday to Sunday, and I'd tell him we had plans (because we did!), and the DC weren't going to cancel them because he decided to put on his dad of the year act.
What also helped was that he has an older DC that he wanted to co-ordinate weekend drop-ins with, and he found that crossover wasn't happening as often as he would like (plus it meant 2 DC weekends in a row for him) if he didn't plan in advance.
Initially you will end up having the DC the vast majority of the time, but if you hold firm for long enough, your EX will realise you are serious about the need to plan, and fall in line.
Agreement over holidays is still work in progress, as I'm about to start mediation regarding holidays (EX doesn't want anything tying his time down 🙄🙄🙄), but I'm hoping to secure more than we've got presently.
Good luck OP.