I think it's hard to make the case for withdrawing a child without coming across as perhaps a bit snobby. But I fully understand what OP is trying to explain.
I have been in almost exactly the same situation and I believe that if the parent feels uncommfortable in the situation, that will make the child wary too.
I sent Ds to a preschool on an estate, I hated it, I didn't feel I fitted in with the parents (having to alter my speaking voice so as not to come across as posh - and yes, I do think I was being judged) and the fact a lot of the kids were from unhappy, deprived families and not treated very well did mean that they took out their frustrations and anger on the other children. I would have thought this was obvious. I took him out and we tried a different one, again with a mix of kids but a much happier, calmer atmosphere and staff I trusted more. I was happier, he was happier.
I didn't send him to the equally (if not moreso) deprived school, either...even though it is 'up and coming'. The thing is, they can put money into a school like that till kingdom come, but it won't alter the fact that many of the kids are just miserable.
It seems a shame to say but I think often poverty and unhappiness go hand in hand. That's not being snobby but realistic.
I knew that my son had enough disadvantages at home, being part of a single parent family, with a sometimes very depressed mum, he needed all the help he could get, and I would rather he went to the more 'middle class' school where he is now and the kids are generally in the happier range, than the one where we used to pass every day and he would be teased and laughed at.
I realise that the parents at his current school may see him as one of the less desirable children from one of the less successful families, but I know several parents who sent their kids to the less middle class school to give them more 'life experience'! Very bizarre. It's probably a question of what you are happy with.
I know the school I didn't favour reminded me of my own school experience, being picked on etc. for being 'posh' - fgs we lived in a council house, but my parents were educated so we talked more clearly than most of the kids
Rant over, sorry...