At 4, my boys did swimming twice a week (one hour lesson and one session just with me), they started swimming when they were newborns as it’s really important to us for various reasons. The youngest (October born so one of the oldest in his school year) also went to preschool / childminder while I worked whereas the eldest started school almost straight after his fourth birthday. They did rugby on a Sunday morning from 3 years old as well, but that was it in the way of clubs. I used to take them running or bike riding straight after school / preschool most days, and they had a pony to ride a few times a week too. we live on a farm so they spent a lot of time outside just helping us with the animals etc at that age.
however, we have 2 boys very close in age who are very similar in their interests (both sports mad), so it’s a bit different to a singleton. Personally I think it’s fine to do lots of clubs, but maybe try to prioritise some unstructured free play time with other children too - so just taking him to the park to play with kids he doesn't know for an hour regularly - that will build those social skills and friend-making ability so much more than structured teacher-led classes and nursery.
You’ll know your own child best and what he enjoys - we always said we weren’t going to over schedule our kids and wanted them to spend most of their free time playing and not having structured classes, and then as they got older it turned out that all they want to do with their spare time is sports training! They are older now (tween & young teen) and both compete nationally in 2 sports, and regionally in 2 more. We probably spend about £250 per child a month including their club subscription fees (competitive swim club for example is £120 a month for 4 swim sessions a week), and weekly training fees where that’s not already included (endurance running club has a small annual membership fee to the running club and is then £5 per training session up to 3 a week). My husband’s training costs are also about £200 a month and I have horses (and swim competitively) so that’s just a whole extra money pit! Our kids by their own choice, do each week: 1x rugby (one boy only) 2x athletics, 3xsurf life saving pool swimming (or in the summer it’s sea swimming), 3x swim club training (other boy only), 1x triathlon specific training, and 1x ski training. They also play football / rugby endlessly at home, are always wanting to go surfing or out on their bikes etc and at weekends we don’t have any training scheduled usually but one has a rugby match most Sundays, and about once a month they have a cross country league race and a triathlon to go to. This is without all their school sports (rugby, football, basketball, cross country, athletics, cricket, surfing, golf and swimming). It’s a lot, but we are a very sporty family (DH and I both compete in swimming, running, and triathlon), and as they get older they do later training sessions or early morning ones so it becomes easier in some ways to fit more in to the days. I’d far rather they do what they love than sit around on a games console (we don’t even own one as they aren’t interested) like some of their friends seem to.