I won't go into the whole story here, but I had a similar problem with my DS in his second year at school.
He too was very bright, could read fluently on arrival at school, worked out how to add and subtract negative numbers in Reception etc. He did OK in Reception - it was a mixed Reception / Year 1 class, he was taught with the Year 1s and given extension work, his teacher was interested in him, etc. The odd bullying incident but that was sorted and he and we were happy.
In Year 1 - mixed Year 1/2 class this time - he became very stressed, very withdrawn, developed a severe speech impediment, stopped enjoying school and started to refuse to go in. He also stopped making progress. his teacher kept the small number of Year 1s together, so he was no longer being extended or working with children of similar ability. he was also (though we didn't find this out till later) being physically bullied and the classroom was sometimes intimidatingly chaotic due to other children's behaviour.
Like you, we deliberated long and hard about what to do. I deregistered him from school to home ed on a temporary basis - keeping to the National Curriculum, but pitching his work at the correct and interesting level, giving him lots of love and nurture and space to recover, and keeping up all the out of school social activities he enjoyed - swimming, football, rugby, Beavers - to maintain social contact with his peers.
We then moved to a different town, and after a very careful process of visiting and evaluating schools (looking for one which was nurturing but stretching academically) he re-entered school for the last few weeks of Year 1. He LOVED it, quickly collected a large gang of friends and has bounced into school every day since (he's now in Year 5). His speech impediment has disappeared, although he still needs structure in his life he is no longer frequently anxious, and most importantly he has a peer group of children with whom he feels quite comfortable to excel academically.
So one option might be temporary home education while looking to return your son to a different school where his particular academic needs might be better met and where there might be a peer group of children more akin to him (I should have said that DS's first school was a small village one, his new school a large 2 form entry town-centre school - at the very extremes of ability there is more chance of meeting another child of similar ability if the cohort is large)? I know that in Scotland attending schools 'out of catchment' is perhaps less common than in many parts of England, but is that a possibility for you?