I spent most summers, growing up, in the water. First with floaters, then I started being taught how to swim by family members.
When I went to school, my parents enrolled me in swimming lessons.
I ABSOLUTELY hated it, and would cry and refuse to go and refuse to participate in the lesson at all, even though I was pretty confident in water and could swim.
My parents didnt insist on the lessons past that 1st year, and I continued to be a strong swimmer (not technically, but always safe and confident in water).
I still hated swimming lessons when I had them with school, years later, Im still not a technically great swimmer. But I can go from A to B and jump and splash around safely, even if I cant stand!
my point it, like PP have said, go for the fun swimming, with you, and less for the regimented swimming lessons!