I think if my child liked the swimming and was prepared to go and it was a usual part of their routine I would continue. It's great to learn to swim, as quickly as possible, to the highest level you can, as young as you can.
If he's going along with it happily and enjoying it, that can be priceless because you never know when children will just up and say they won't do it any more... same goes for music lessons or other sports. So if it ain't broke I wouldn't fix it. Also if you stop a routine it can be harder to restart one than just carry on.
And bear in mind it's really hard to get them that full-energy running around time in the winter when the weather is bad, so a guaranteed block of physical exercise weekly is a good thing. Plus if your pool is like ours, if you stop, you have to join the waiting list again and then frankly you might as well just give up on his lessons and restart by getting pregnant with a different child because the wait time is 100 years!!
However, if the cold is something he massively complains about already, and you're thinking as it gets colder he might just say no no no I don't want to go, because of it, and then be put off forever, then a tactical breezy "let's do a different sport in the winter months" might be prudent.