You have my sympathy! It's a really difficult decision.
At our primary school there were generally one or two children each year who couldn't go because of cost and often one who won't go because they can't face it.
My son was very reluctant to go in year 5 - he's never liked sleepovers and has always had difficulty getting to sleep without the radio which of course he wasn't allowed to take. He never really did cuddly toys so that didn't help.
We spent months weaning him off the radio though he still wouldn't do a sleepover, and made him go. He loved the days - the teachers all thought he had a terrific time because he did enjoy the activities - however he absolutely hated the nights but didn't want to say to anyone.
In year 6, he started getting stressed in October about having to go again and was getting very panic-y by February to the point where it was a constant factor in his brain. To some extent it was a matter of his level of trust in the boys he was sharing with and his teachers. The first year there had been boys in the year above who he knew he could rely on to be kind.
In the end we decided that having been once, it was his choice about whether to go again. He didn't go and instead had 4 days with me doing things like visiting grandparents, going to galleries in London, volunteering in a local wood and one day in school helping get the set ready for the school play.
If we'd done it the other way around, perhaps let him miss out the first time, perhaps he'd have been determined to make it in year 6. Who knows.
Anyway, he's off next week for a two-night trip with his year 7 group and is being very grown up about it. He doesn't expect to like it at night but is willing to take the hit this time and that's been his choice.
There isn't a right answer really - whatever you decide will have a positive side and negative so you can't make the wrong decision :-)