Oh, god, how awful and tactless of them. My ILs planned to come and stay with us at our home, tacking us on to the end of their lovely holiday as a couple in Croatia, and I said, no, we will meet you in France, we will have much more fun if we are all on holiday together, exploring a place that is new to all of us. Perhaps they thought this inhospitable and unkind, but having hosted them many times before, I wasn't willing to do so again, and be chief cook, bottlewasher and tour guide! I was polite and straightforward, they accepted it, and I am now looking forward to seeing them, not dreading it!
I would urge you to be similarly straightforward with your ILs. "Sorry, we seem to have given you the wrong impression about having guests on holiday with us. This particular week is a much needed break for the four of us, and we want to relax just the four of us, after a hard old slog of a year. But we are looking very much to seeing you when you are visiting later in the year, as planned, and that will be lovely when we're all recharged by this break." Or something on those lines.
They shouldn't have barged in, and they shouldn't need telling, but since they have, and they do, you mustn't feel bad about sticking up for your holiday!