I took my DS swimming for the first time last week; he's nineteen months old. I didn't avoid the classes because of worrying; we simply didn't have the money. It would have been nice to experience it I guess, but DS reacted wonderfully to his first time in 'deep' water. He loves splashing in his bath, but always squirms when we wash his hair. I was hoping that swimming would help that, now that I've finally been able to afford a swim suit which fits (damned boobs!).
He was quite nervous at first, but my dad was there as well as myself, and soon he was splashing merrily away with a small float. He 'fell' several times as he got a little too confident, and went partially under, swallowing a little water because he doesn't know to close his mouth, but otherwise coming to no harm. By the end of the session he seemed perfectly alright with the water going over his face (normally he doen't like this), and not long before we left, he was being over enthusiastic and went completely under for a moment. When he came up he looked surprised, but not at all bothered. I took him again today, though unfortunately to a different pool, so he was a little dioriented. He loved it again though after he got used to the place.
It doesn't seem to have made a negative impact on his time in the water that I didn't do the classes; he took to it like he was a natural, going flat on his tummy with his head up as I 'flew' him through the water. This week he even seemed to be trying a bit of doggy paddle!
I guess what I'm saying is that even if the classes do help the children who cope with them, if you don't, they aren't automatically going to panic in the water when you take them for the first time. And given your experiences, aligriff, I would definitely say you got out in time. You know your child, and if it makes him distressed, it certainly isn't worth it.