Mmm... tough. My first proper residential was Yr 6, but we did "school sleepovers" from Yr 3.
3 nights is quite a long time, but it will depend on the residential. Some trips will be camping less than 1 hour from the school and will include a "family day" so there are fewer worries.
Problems that would make me think twice about the trip:
-child wets the bed
-child repeatedly states they don't want to go
-child does not enjoy sleepovers with friends or grandparents
-residential is more than 2 hours away
-children have had trouble in the past
-he's bullied by his classmates
Since he's only said he doesn't want to go once, I wouldn't worry. At 14 (Yr 10) I broke down and told my parents I didn't want to go on an exchange to Japan, but I absolutely loved it and always wanted to go but was just freaked out but an older girl's tale about a drama involving getting lost in Yokohama.
Do make sure he knows you don't expect him to go. He might be trying to put on a brave face.
Anyway, that's just the trip being in Yr 3. The change-without-consulting-the-parents, the cost and the fees rising is a totally different kettle of fish!