I'm speaking mainly from my personal experience here. Developing socially is much more easily done if you have common ground with a group. Most bright children can thrive in a typical school. But a minority of bright children will have very little in common with even the brightest in an average mixed class. I strongly agree with someone above mentioning breadth and depth as most important in educating academically gifted youngsters - and where you might achieve that in a school setting for the 1% or even the 0.1%, the 0.01% need something rather different to approach achieving their potential. Only some of that 0.01% get that opportunity.
I was probably in the 0.01%. My school experience was miserable at times and boring at others. My potential was not nearly tapped although my parents did a lot to encourage my interests, and near the end of school I achieved a lot through olympiad training. Apart from that, I didn't learn how to knuckle down and work steadily - my last minute work regularly got top marks. Last minute continued into university and I didn't really start to run into problems until well into my 2nd year. Crashed and burned in my 4th year yet still did well enough for a solid 1st class degree. A different earlier educational path could have made a difference. I eventually discovered I was a people person - at the age of 22, having always thought I must be an introvert based on how I interacted with most people I knew. Once I was at university I had a more genuine peer group in which to become (somewhat) socially skilled.
So a question is how can you provide the right opportunities to the 0.01%? Highly selective schools can sometimes do it, but can still suffer from being curriculum bound. Boarding would be necessary to get sufficient numbers in one place except in the largest cities. Nowadays the ability to provide real enrichment material via the internet can be a game changer for these children; so I think I'd advocate for short-term residential activities supplemented by online challenges. Meeting and getting to know others "like yourself" can really help children develop self-confidence and a positive identity.
We are very privileged to have been able to obtain subsidised boarding education for our youngest that puts him in an environment with others of very high ability - his progress socially in just a year of that environment is amazing. His sister would have benefited from something similar but I suspect the environment created when you put a group of very bright girls together is somewhat different, so she is in a different sort of school. And socially still feeling fairly isolated though she is not without friends. It is hard to tell but I would guess both are more the 0.1% than the 0.01%. She is as far as I can tell, being realistically challenged in her favoured humanities subjects; has been pushed somewhat in maths; but in sciences her relative disinterest has been strengthened by the boredom of GCSE work (sitting exams this year). Her brother's school, as I understand it, will take them beyond GCSE work then ensure they are ready for the actual exam content near the time.
For me, in the place I grew up, acceleration and permission, as a girl, to attend one of the top boys' schools might just have done the trick ... but both options were as unfeasible as each other at that time and place. A few years after I finished, acceleration became quite fashionable for a while - but as they were trying (to be "fair") to accelerate not the 0.01% nor the 0.1%, but more like the top 5%, the demands of such a system were too great for the majority of children offered it, and I doubt it took long before people could turn around and say, "See, we told you, acceleration is not a good idea".
So yes, for me a "school for the gifted", even if it had been a long way from home, might have been great even as young as 8 or 9.