I used to wait for DH to say no, he was never able to do it effectively, either putting it off until the last minute or not explaining properly.
I think you have to nip in the bud yourself right now. You are a couple and asking one half to come on holiday isn't on. its also inviting themselves behind your back which is a bloody cheek and there's a reason why they asked DH and not you, because they knew he'd find it harder to say no.
I think you have to be the Bad Guy, take the hit and make sure you ring fence your holidays for yourself. Otherwise they will think its OK to keep asking your DH and jumping in on your plans.
Take courage in both hands and email them directly yourself. Say everything you said in your post. Writing is better so they have time to read and digest. And then prepare DH to have the talk when they've read it. It's perfectly reasonable and if they take offence its on them.
"Hi Inlaws. I hear you've told DH that you want to stay nearby when we are on holiday. I'm so sorry but we planned a quiet relaxing getaway from it all with just the DC and us on our own and we want to stick to that plan.
It's our only week's holiday this year and we've been looking forward to this so much and we don't want to suddenly change or expand our plans now. We've already spent ages talking about exactly how we want to spend this break and don't think you would find the days out and activities we have planned for ourselves suitable for you.
Maybe later in the year, a weekend away planned by all of us together, would be more suitable for everyone instead. "
Weeks away with the DC just on our own were brilliant to have the time and space to really play with them. When they were small their speech really developed during that time. It was also a time when I had DH to myself and we were really able to relax. When we did go away with others it turned into a marathon of cooking, entertaining and endless chatter, which was exhausting.
Stand firm OP and happy holidays!