I would definitely let her go.
If she's in yr6 she will soon be demanding much more indepence, and release from your protectiveness. Having this sort of experience will definitely empower her and give her convidence: it has a very good track record of doing so, that's why schools do these sort of activities.
The activities will be age-appropriate, and if she can't swim/can't ride a bike, then appropriate options will be offered. Staff at these centres are used to handling children of different ages/abilities.
Tbh you do sound rather over-protective, and worried about things which are statistically very unlikely to happen.
If you continue in this way, your dd will pick up on this worry - if she hasn't already, and it will affect her confidence.
Sorry if I sound blunt, but I think you need to loosen up a bit.
My boys have done 'adventure-type' activities in the Lakes every summer since they were 8, and I take yr7s away every year on a foreign trip (not adventure though). The children adore these experiences, and talk about them for years afterwards. If all your dd's friends went on the residential, and she didn't, I think she would be really missing out.