While it may be mostly used by those whose parents are working, is there any reason why your DS cannot have some fun, organized activities and space outside of school with his friends and make some new ones just because you DON'T work??!!
I do work FT, so DD has been going to camps since she started school (crèche all summer before then, apart from our own hols). When we are away, we don't put her into a FT camp, but she usually wants to go to whatever playgroup etc that's available for at least 1 day.
This year, we are away for 2 weeks and she has asked to do surfing camp daily while we are there (place we know well and can't do that here) - 2 hours per day. And when we get back, I have another week off but she has asked to do a cookery camp then, as that is half days (finishes at 2.30) so she can't normally do it and she still has afternoons with me or playing with her friends.
Absolutely chosen by her!!
He will have a ball, if he hates it (which I doubt) you can keep him home, but it is good for him for reasons I put above. And your other 2 DCs will get the benefit too of those extra hours in terms of your attention.
You can always plan something fun together as a family (all DCs I mean) for another week that he is not in the camp. 1 week of the whole hols is not something to feel guilty about.