My DS went to IOW residential in Y4, possibly the same place.
I must admit I had a little wobble when it was first announced. It didn't help that I first heard about it at my first governor meeting as a parent governor and the governors had to agree to the proposed trip! Think it was the fact it included a ferry ride that made it seem so far away 
However, I agreed, along with the other governors that it was a good trip for the children to go on.
The children loved it. I might have been anxious when waving DS off, will admit to a few tears, but I am a very anxious person, and didn't want that fact to spoil DS's and the other pupils chance to have a fun residential.
When DS was slightly younger he went on his first cub camp. At that time he had never had a sleepover as he had never wanted to before. One of his friends was in exactly the same position. So we organised reciprocal sleepovers together with much of their cub camp kit, a few weeks before camp. Loved sleepover and loved cub camp, with no homesick problems.
If your DS is worried about going then it is fine for him not to go, but please don't project your worries on to him.
Why don't you try to arrange a few sleepovers over the holidays and see how he gets on. If he doesn't get on with them then a trip to IOW is probably a trip too far, but if he likes them it may give him confidence to go.