This is just a thought, do ignore it if you think it's not relevant.
DS is scared of water, doesn't want to even try to do anything in the pool.
OP You did say later in the thread (paraphrasing) that he has swimming lessons at home and loves them, so I think the problem here may be the holiday pool and the randomness of the actions of the people in it makes the whole thing for your DS totally opposite to his idea and experience of 'going to the pool' or 'going swimming'
He hates any group activities, trips, beach. Anything remotely fun is a no.
Again, how much of this is outide his experience of those activities with a similar name at home?
Maybe his reluctance is due to his mental image and lived experience of the activity being the thing he does at home and now on holiday, you're using the terminology he understands, but the reality of that experience on holiday is vastly different for him so he's floundering a bit.
Group activities on holiday won't have the people he's used to being with at group activirties at home.
The beach on holiday will look, sound and feel very different to beaches at home.
Things you think of as remotely fun to you, may be very strange and outside his comfort zone to him.