Hi there,
in music it's certainly true that if you don't learn an instrument at an early age it is very hard become very good on it later on (at least this goes for string instruments and the piano, for wind instruments it's a bit different). There are some exeptions, I read about a handful of violinists who started later (with later I mean anything between 10 and 14, certainly not much later than that) and became professionals, but this is rare.
But even if becoming a professional musician is not your child's aspiration, it's still very hard and frustrating to start an instrument late, and quite impossible to reach a high standard (and yes, I do think reaching a rather high standard is more 'fun' even if not being professional).
I started the violin at 7 and the piano at 18, and while being the same 'musical' person starting late was very tough and while violin playing has become a second nature (still requiring practice obviously) I will never ever feel the same with the piano, however much practice I put in.
Obviously a musical person can still enjoy music later in life even if not playing an instrument and if not being exposed to it from a very early age, but it's far more unlikely from my experience.
With regards to swimming, funnily in our case it's exactly the other way round: I could swim aged 5 even though I was never the most sporty person, and my sons still struggle to get their 5m badges aged 6 and 8, although it has to be said I learned breast stroke first which is supposedly easier for 'surviving', they learn front and back crawl first, so it can't really be compared.
I do believe in stimulation and reading to children, singing and introducing them to many things early, however, sometimes I think they need their own time, and a lot is already 'there' (or not). They will do everything at their own pace and according to their ability, and sometimes it's easy to forget this and you think 'they should be doing it by now', then they do it a year later and however much you do, it won't 'click' any earlier in some children.
It's important to try and do everything to hopefully make them reach their potential, bt however much you do, a child with a rather low IQ will never have one of 130.
There are IQ tests which distinguish between the actual IQ and the 'well educated/supported' bits.
Also I don't think that taking them to toddler groups or toddler classes makes them any more confident or sociable at all, otherwise all children who have been to these groups would be fine in that respect, and it's just not the case. It's a trait that's already there, while obviously you do your best to encourage the positive.