I think it's not as simple as that if the lesson is mid-school day.
The school may decide to take him anyway.
I would find it hard to put a child in the position of not having his kit and potentially being told off.
I think I would be very clear, in writing, to the school that he was not to be included in the lesson until this matter was resolved, and then I would turn up at the pool on the day to make sure they abided by my wishes. And to observe the lesson, too.
Though of course it's likely the instructor would be aware you were there and deliberately be very reasonable while you were watching.
This is a difficult grey area, the subject of swimming in schools - on one hand it is a curriculum item, OTOH it requires a parental permission slip. I imaginethere are insurance issues should you decline permission, and the school would be on shaky ground if they ignored your wishes, particularly when you have such strong reasons for them.
That's what I'd do. If you email tomorrow morning, it's in writing, and they will have to respond before the lesson if they want him to attend - so it sort of puts the ball in their court.
I hope you find a way to resolve this.