I was in a similar position. I have two DDs. I had my oldest in a small private prep school for nursery, reception and year one and two.
She was "ok" but the class was boy-heavy and very old fashioned and strict.
her friends weren't real friends but were sometimes nice and sometimes not.
DD was anxious and pressured all the time.
I also had a younger child...luckily I managed to get DD1 into a neighbouring village's outstanding state primary and slowly but surely she began to thrive.
It wasn't plain sailing.
The first year there (year 3) she was anxious....wouldn't really join in.
It took till year 4 for her to really settle. I chose that school because it fed into what I thought was a good secondary...turns out it wasn't andwe moved to Australia!
She went into a new middle school here in Oz aged 11 and again thrived..this time immediately.
When you have to make choices like this you get SO wound up because you're trying to make "the right" choice and there IS no right choice.
If you don't move him you might find things don't improve and every little bump will make you question your choice.
If you DO move him and he doesn't settle in the new school....again...you'll kick yourself that you should have stayed put.
So...reconcile yourself that there's NO right answer.
Here's my thoughts.
You have a great house, an allotment and good friends.
He's a worrier and anxious.
He's still going to be those things in the new school.
Have you checked out how supportive the new primary is for anxious children? What's their ofsted like?
The good secondary might just tip it for me...what's the secondary choice like where you currently live?
And finally...what steps have you taken to help DS? Have you organised playdates? Got him into cubs?