Some words of experience:
I was identified as Highly Gifted at age 7 (lots of things I was excelling at, and a high IQ test).
My (immensely proud) parents moved me from my local (socially deprived area) school to another school (in posh area) with specialist programmes for the Gifted.
And it was pants. The extension work didn't interest me much but much worse, I was badly bullied and only bumbled along academically. I was miserable for years; it took years after I left for me to recover and finally start to excel academically again.
Is your son happy at his current school? Because based on my own personal experience, I would not upset that.
I currently have the problem that DS1 is excelling in spite of hating going to the school -- social issues partly to blame. It's tricky because DD is happy at same school, and I can't see how DS1 would be that different at another primary.