Its lovely that you are so keen to give your child so much attention, enjoyment, support and opportunity. This is an attitude that will truly benefit your child in the long run.
However you are misguided to take swimming so seriously at this stage. A bit of splashing around in the toddler pool is just as much fun for him, and safer and more convenient for other pool users too. You can sit or walk on your knees a bit if you want to pull him.
It is really, really ok if the summer holidays "sets him back" a bit anyway, all children get set back a bit every summer break, and you know what, the parts of their education they have to learn twice are often the bits they remember most securely and confidently 5 years later!
Particularly swimming, it will be fine, really. In a few years time it will not make any difference at all whether he goes swimming every week or twice a year at this age. If he is good at swimming, and enjos it, he will work at it and do well. Howeveer, like many of us, you are likely to spend hundreds of pounds and feel great pride in progress for years, then find you hit an age, earlier or later, when the child has just had enough, and wants to put their time and energy into something else.
Thats fine too, DS1 was an "advanced swimmer" at 5, but prefered basketball at 10, and now, at 15, swims occasionally for a bit of fun on holiday, nothing more.
That's fine though, he can still swim, not so well as he did at 5! but he enjoys it occasionally, he has happy memories ( and badges and awards) from swimming when he was little, and is still good friends with other swimmers he met when they were little, even toddlers.
In the end, you want a happy, well rounded child, who feels confident that he has your support in everything, and I can see your child is going to have this, without you having to worry about his progress in swimming at 20 weeks.