I suppose it depends totally on the child and how comfortable/confident they are with other people.
I am a Brownie leader, and we take our girls away at 7 if they want to come. Almost all do, and I feel a bit sorry for the ones who don't, as they miss out on what all the other girls think is the best thing we do each year.
I do think that as they start to grow up, getting used to this kind of thing graudally from a fairly young age is a good thing. Sleepovers are just one night in someone's house with their parents there, Brownie holiday is 2-3 nights, in an unfamiliar location, not necessarily like a home, with 20 other girls, Guide camp might be in tents, school trips away etc. might be for 5 days by the time they are 10-11ish, and going away on trips abroad for at least a week, with e.g. Guides etc. by the age of 14 is not unusual. There are very few children who would be comfortable with the last scenario if they've not been 'acclimatised' with the earlier ones at a younger age.
Nobody has to do any of these things, but the risk of missing out on important experiences later on will probably increase if they have never got used to staying away at a younger age.
D