As so many others have said, it depends what you mean by smaller.
I'd prefer the dc all had their own bedroom
I'd prefer (when they were small) we had a decent garden
I needed two living rooms
I needed a kitchen big enough for a table we could all comfortably sit round.
I needed a space to wfh.
So, no, I wouldn't 'lose' any of those to have two holidays a year rather than one.
But then I wouldn't have stretched myself to get 'spare bedrooms' or a garden room or anything else 'extra' if it meant not having holidays and having to say 'no' when invited to special events.
But, OTOH now the dc are grown, I am actively looking for a smaller house, smaller garden, etc, which will free up money for a few 'big holidays' before we get too old to travel much, yes. But that is only one reason for wanting to move - I wouldn't go through all the hassle of moving "just" to free up money for luxuries.