Why would you want to have such a horrible, avoidable, needless mess when all you need to do is say, sure, we can do that? You do know the single most damaging thing for a child is conflict between parents, yes? So why are you going out of your way to create it?
I don't get it. You talk about being child-centred, and then refuse absolutely to try to make the most important relationship any two adults in his life can have as cordial as possible. Why? Is there a back story here? (One that justifies this, I mean, and whatever happened between the two of you doesn't, short of abuse.)
There is no court order that forces a parent to see their child, for the very simple reason that a parent so shit they needed one would be really unlikely to make that child feel good in any such contact.
And you seem to be under the impression that you can just rock up to court and get a child arrangements order, or whatever they call them these days. You can't. Mediation is mandatory first, and you'd need something better than, well, he wanted a swap for a single weekend, two years running, because there is something called the No Order Principle. A judge can only make an order relating to a child if they are confident such an order is better for that child than none. Here, you have a contact arrangement that seems to work, and you're having a hissy over a single request to alter a single weekend - why? I really don't get it.
Does he pay child support okay? Honestly, I'm trying to establish why you are so angry. You've written nothing that's bad about him at all. The fact you're so bitter over a single request indicates he's incredibly reliable the rest of the year.
Do you never have a wedding or a party or anything you would like to go to child-free? Why not just agree to this, and say you'd like the favour returned some time? Again, I just don't see where the issue lies. And for the record, I can't remember the last time I didn't support a mother in a contact issue. Most seem to have men who never turn up, don't pay child support, sneer at the mum and/or try to insist their new partner is called Mummy... and you're angry he wants to swap a weekend for extremely good reasons, once a year.
It's just... weird.