My DS (now YR6) had a very mixed time with school swimming lessons in YR4/5. He also has ASD. The first thing I would recommend you do is speak to his class teacher as soon as possible.
They will need to know about his possible reaction for their risk assessment and also to think about if they're going to need to take an additional adult from school. (I assume the school know about his diagnosis? You say his classmates don't.)
It's most likely that the swimming lessons will be taken by a professional swim instructor. Some are better with children with additional needs than others. With my DS, his teacher told the swim instructor about his ASD but my DS would still have meltdowns in most lessons. Sometimes he'd have to sit on the side of the pool he was crying so much. I think he only had to actually leave the lesson once though.
BUT there was also another boy with ASD in my son's class and he absolutely loved the lessons - so it's really hard to tell how children will react. As you say, he may be completely fine and really enjoy them.
The teacher wasn't great with him but I guess to be fair it must be tough. They've got around 25mins once a week to teach these children to swim. My son's class was divided into 3 groups: the strong swimmers were supervised by the class teacher and then the other two groups (one who could swim a bit and one who couldn't swim at all) were taken by the instructor.
In terms of opting out of the lessons, I think that's going to be hard. I think they are compulsory - but as he has a diagnosis, the school and the swimming instructor - have to make reasonably adjustments by law for your DS.
If he doesn't attend, the school will have to find a class for him to sit in for the half the morning while the others are swimming. Which will obviously make him stand out.
If I were you, I'd be tempted to give it a go. If he really hates the lessons, then you could speak to the school about trying to opt him out.
As an aside, I took my son to a specialist ASD swim class outside of school. It took him around 3 years to learn to swim properly but he absolutely loves it now. I think many people with autism enjoy the sensory experience of swimming so hopefully your son will master swimming and come to love it too.