I'm of an age that we didn't do residential school trips - even at senior school. There were just ski-ing trips which relatively few pupils went on - trips out were always just day trips.
DS1 is at a junior school where they do a school trip each year - building up from 1 night to 4 nights as they get older. DS would quite like to go - but both dh and I don't agree with school trips before secondary school so he hasn't been. DS has resigned himself to the fact.
What really annoys me though is the way the school treats those that get left behind. On the one hand they are saying that they are learning vital skills and doing amazing activities that they need to do to enhance their lives and be better people - but they don't give those staying behind the chance to do any of them. They do run a different curriculum and have a topic to work to but quite frankly it's pants in comparison to the things that the kids are doing on the trip.
The thing is - it's a big school with lots of kids. Even if they wanted to, there aren't enough spaces for everyone to go - there's always a rider on the bottom of the application form along the lines of 'there will be a ballot for places for those that get their application in'. This means that there's a small but significant number that don't get to go on the residential (between 1/3 and 2/5 I reckon) - enough for one to two classes.
This means they know they are depriving the ones that get left behind. There are lots of great centres around locally where they could do all the activities and more that the residential course offers. Even if we had to pay (that's not the reason I object to the residentials) I'd be happy to pay towards covering the cost of it. But they don't bother too. It would be a bit different if it was a whole of year trip and there were only a couple of them that didn't go - I can see it wouldn't be practical. But when so many of them don't get to go - I just think it's bad. 

Those that get left behind do suffer when the others come back too - they are definitely made to feel they missed out, the friendship groups evolve a bit and so on.
Which is all a very long way round (sorry!) of saying - I wouldn't force ds to go if he didn't want to - but at the same time, I'd want to make sure that he got something positive out of being left behind and that the school provided him with opportunities that he would have otherwise missed out on.