DS was ill from the Friday before October half-term to the Monday afterwards. (I'm sure some people at school wondered/doubted, but yes he was honestly just ill.)
Afterwards we were very sorry to not have had a break (being ill is not really good for recovering/resting from school stress).
Later, in the Easter holidays, we stayed at home too, but had guests staying for the whole two weeks. DS was absolutely exhausted by the end of the hols. It would have been better for him, had we gone away to a quiet beach or something.
Booking a holiday away can be very restful, or not. Staying at home can be very restful, or not. I think the question is not 'should we go away?' but rather 'what should we do to allow DC to recharge batteries/to allow us as a family some down-time, time to re-connect'? And that depends a lot on your family, how you are together, what you like to do.
That said, I would NOT plan for exhausting travelling to happen on the Sunday before term starts again.