I would say it's your house, and so it's up to you who you have to stay and what conditions.
You're allowed to set boundaries, just as much as anyone else. :)
You have the right to protect your emotions and worries ect by setting a boundary and saying no.
Just as much as if you asked to stay at your sons whilst he was away, and he said no.
He would have a right to say no too.
Okay you may be a bit disappointed. And in your current situation he may be disappointed.
But in summary he is asking for something from you, and you are fully justified in not giving it.
However, I would add, that if it was more of a need and not a want for him to stay at yours then I'd consider it a bit more thoroughly. Ie in terms of pushing what you're comfortable with.
In this, I mean instances where he would be suffering ie homeless, or not well with broken legs and needs somewhere to stay, or his work means he will need to be staying in the area. And he usually lives far away so it would cost him £100s to find somewhere else. In the latter he could offer to pay you some of the costs.
But i think it's important to set boundaries, otherwise when you do give, it will be expected rather than appreciated.
(my relatives are currently going through issues where they had failed to set boundaries and now it's causing much bigger problems)
In your situation, depending on how much anxiety you'd have, and how much you want your son to be happy vs your inner peace. You could offer to meet halfway.
Ie say he can stay if he does a specific list of cleaning jobs whilst your away, the state you expect the house to be in when you get back, and that if this is not done it means in the future you will not allow him to stay whilst you're away on holiday.
Ie a chance to prove himself to be considerate and responsible, rather than going off your worries which are not yet proven to be true or untrue. (as far as I am aware).
It would however probably cause you to loose some inner peace whilst you're on holiday.
And he would probably be the one gaining a lot more in the situation.
But the other thing is not about him staying, but how loved/close he feels to you and generally supported. And I would say you could also consider how you want him to feel that way.
:)