This.
The leaders will do events based on the permits they have too. If they have a swimming permit they will lean to activities like that because that's the adventure stuff they have the skills in. Learning a new skill and getting permitted in it is a huge commitment which goes way beyond simply being a leader which is a big enough commitment in its own right.
Why should other kids lose out on that activity because of yours? It shouldn't be a race to the bottom either.
Being inclusive can just mean it will be giving the opportunity for kids to do that activity who would not normally get that chance - because they wouldn't have thought of it and don't have adults to enable it.
YOU have made the decision he can't take part, not the leaders though. You've assessed it's too risky. That's a lack of trust in the leaders - who will have to do a bunch of risk assessments to do anything with the kids.
With the correct supervision and equipment even weak swimmers should be fine. If they are surfing they will not be allowed in the sea without buoyancy aids and sufficient supervision.
As for doing rock climbing. They either would probably have to pay for several instructors (which gets very expensive very quickly and thus is much more excluding than surfing) or have a rock climbing permit.
DH has spent about three years jumping through ridiculous district hoops to get his, even though he is a very experienced and competent climber. More so than others who had permits. It's not an easy process.