Oh dear.
"I can't permit that..."
That comment rather gives you away - this is about control.
Bad news for you - it isn't your place to "permit" or deny anything, when it comes to what he does with his son on his time. I'm saying "his son" quite deliberately - because you seem to have forgotten that he is just as important as you. You are both the boy's parents. You are both equal.
I'll say it again. You. Are. Equal.
You don't outrank him. He doesn't need your permission. You don't get to say what does and doesn't happen.
The law is clear. The boy's father can take him anywhere in England or Wales during his time. So can you. Because you are equal. With no court order in place, Dad would need your permission to take the child out of the country. And you would need his. Because you're equal.
Beyond that, it's a question of co-parenting. Ideally, you would both tell the other if you're taking the child away - for practical reasons, in case of an accident. That means you, as well as him. Because you're equal. It isn't about either of you getting any sort of say in those trips by the other - it's just a courtesy, and it runs both ways.
Your tone all sounds very controlling. It is all about your role as the "present parent". If that is the tone you take with him, I'm not surprised he resists it. It's human nature. When someone constantly acts as if they're superior, many people will resist that. If you start treating him as an equal, then you may find things change. If you demand that he keeps you informed as to his movements, then you must do the same for him. And if that feels like an intrusion on your life, then it's an intrusion on his too.
I know it's not what you want to hear, but from the tone of your posts I think you could get a long way by improving your attitude to co-parenting.