@mummy917
The thing to remember is that many times when a person (him) accuses another person (you) of being controlling, it's not because they are. It's so that person (you) bends over backwards to show how UNcontrolling they are so the other person (him) always gets their way. This is a very well known manipulative technique. And the only thing it 'proves' to them is that the tactic works, since they know you aren't really controlling in the first place.
So you need to really give thought to all the things you're doing to accommodate what he says he wants so be sure you aren't actually doing yourself a 'wrong' to try to do him a 'right' simply to show him how 'uncontrolling' you are. You aren't an unreasonable person and you want what is best for your DC. Let that guide you. Just remember that you need to prove nothing to him. He says you're controlling? So what? YOU know the truth.
You don't have to say, but did you speak to the solicitor about the benefits of a 'spelt out' court order wrt the children vs an 'informal' agreement re access and/or maintenance? Both have their plusses and minuses.
My BFF ended up having to get a court order that spelt things out down to a gnat's arse because her ex kept jacking her around on access in order to disrupt (control) her life and her plans. Not showing up with no notice or showing up on a different day because he arbitrarily decided to change his weekend, showing up or dropping off early/late, or not dropping off at all so she had to go get the child. She even had the order amended so that each had to give a minimum 72 hour 'notice of change' to change dates AND that the original date was then 'cancelled'. She also had '30 minute period' put on the order so he wasn't to show up more than 30 minute early and if he didn't show up within 30 minutes after of his pick up time, she (and DC) was free to get on with her day and that weekend's visit was cancelled.
Sound controlling? No, just keeping her sanity against HIS control. Now, this order was over 30 years ago and in the US. No idea if such a specific order would be written today. But it just goes to show that not everything that appears to be 'controlling' really is.
So, just put yourself and DC first, as you are doing. What your stbx wants or thinks is irrelevant. Be guided by your solicitor as to what is reasonable or not.