If you have said no already then I would just remind that you have said no and then change the subject.
So if they say, oh we are going camping next month, we'd love it if you come with us, you can just say "oh you know how we feel about holidays with friends..."
If you have to tell someone, just tell them! "In theory I am sure it would be lovely, but we prefer to keep it just the 4 of us" That is all you have to say about it, you don't need to explain, do not fill the silence with lots of excuses!
Yes, as you say, it´s flattering and I feel somehow that there´s something wrong with me that everyone else seems to want to spend so much time together and I am the only anti-social one. On the other hand, DH feels the same way and I don´t want to force him to spend his precious holiday (he really doesn´t have much and most of it is spent caring for his aged father) with friends, if he just wants quiet time to him/ourself/selves.
I´m really not good at saying no and I don´t want to hurt their feelings - as I wrote, the last time we said "no" really loud and clear, the friends stopped speaking to us completely. I have tried explaining it to them gently (whilst saying "we want to keep our holidays just for the four of us"), but they seem to think, if they keep pestering, we´ll change our minds one year... The problem is that I find it really painful to say no to them and I hate having to do it time and time again. I just don´t know how to get the message across, that I just want them to stop mentioning it.
We have several different lots of friends who have wanted to go away with us for years. One particular family suggests every year for us to come with them on the main summer holiday. Plus they would like to go for weekends away with us. If I mention that we are going away, they make a half-joke asking in a half-offended way why we didn´t invite them. I have tried explaining to them that we really value the time together just the four of us (me, dh and dcs) and that I am a person who needs more private time than most people - either on my own or just with the dcs and dh. They just never drop the subject. Another family ended up not speaking to us for 6 months after we said we didn´t want to go camping with them for a long weekend. More recently we gave in and went away with them for a short weekend. As soon as the others found out about this they said that now we had to agree to go away with them too. Aaarrrgghhh. Now I wish we had never been away with the one lot. How can we remain friends but not have them asking us all the time??? (I suppose just go away with them is the answer, but as these friends are in other areas of life, if we do go away with them they will expect that this is now a regular routine).