Maybe 3 isn't too young for a child who's natural inclination is to take instruction, or a child who really really wants to learn to swim....but for a 3 yr old to squish their natural inclinations and fit in just because they're supposed to is a bit ambitious, IME
I can totally relate to the worries, though. My dd started dance lessons, at her request (bf was doing them) at just 3, and was utterly miserable after the first one or two, when the novelty had worn off. She disrupted the class by getting upset at various instructions, wanting to do her own dance and not what she was told, etc. I felt just as you did, and worried...wondered if I wasn't strict enough/too strict and so on. Honestly, it makes me now to think how much I agonized, I suppose I looked at all the well behaved little girls in blue tutus and their smiling mothers and felt I'd done it all wrong! This in spite of the fact that she was never a 'problem' at pre-school, fit in well, and so on.
With hindsight, I can see quite clearly that she was too young to conform/obey in a context that, to her, meant fun and self-expression, and being with her pal. Learning dance steps and following the class made no sense to her, no matter how much we role played or talked it through.
As soon as I stopped feeling responsible and ashamed (if I'm honest), and stopped seeing her refusal to be taught dance as a negative, or my fault, I pulled her out of classes and went back to letting her have fun dancing to music at home. And started to see that her refusal to join in wasn't necessarily negative at all.
I asked her at 3.5 if she wanted to learn to swim (she loves splashing around in a pool) and she actually said 'maybe when I'm a bit older'. She knew she wasn't ready. Now she's nearly 4, and has asked for lessons - so after Xmas, we'll give it a go
HTH, your ds sounds lovely btw.