I don't feel that children need to go to nursery at all. Remember back to your own childhood - did you go to nursery?
However, do you really mean nursery, or do you mean some kind of pre-school group (you say you already go to groups).
Personally as a child, I went to a playgroup a couple of mornings a week (as my mum helped to run it). At that time, I think playgroup only accepted children aged 3 to 5. Back in the 70's, playgroups were play focused (Learning Though Play) and that I feel should still be how groups are run... children learning to play with other children, getting the opportunity to play with things that they may not at home, eg. sand/water/wet-sand, painting.
I care for a 3 year old, and he does not go to nursery (and neither did his older sister, who recently started at school). Instead we enjoy being out and about, exploring the world around us, visiting places (castle, farm, sea shore, science discovery centre, going on trains/bus/boat) and getting exercise (country walks, swimming, play gym). He is an IT expert (well not as good as his older sister, who seems to be top of the class at school for IT skills), as at home the children have access to a computer and thus practice mouse skills, do language, shape, matching games etc.
We walk frequently with some childminder friends of mine, so all the children get to interact, form friendships, fall-out over minor things, etc.
So there is no need for children to go to nursery, if the parent/carer is giving the child opportunities to interact, to see the world around them, learning things like crossing the road and simple maths (money, sharing out objects, counting objects).
Go with your gut feelings, you know your son best. Don't be pressured by others. Government only provides funding for a pre-school place from the term following a child's 3rd birthday, so no point sending them before then... and even then you may feel your son is better being with you.