My son is at a special school (he has never hit someone btw) so this doesn't apply to me.
But, I think there are quite a few reasons sime parents don't want special schools.
One is there isn't always actually a school that is the right type. The special schools in your county might be for severe learning disabilities and only offer life skills. If your child is able to do gcses, it's quite hard to send them somewhere where they are teaching teens to button shirts. Special schools availability just isn't even across the country.
The other fear is if your child is prone to impulsive behaviour or copy-cat behaviour, the last thing you feel you want is to send them to an semh school where they can pick up all sorts of new ideas of how to misbehave, and you know that county lines recruit at the school gate. You feel hopeful being around better behaved chikdren will rub off.
The other is special schools are boy heavy. If you have a daughter it's quite daunting sending them to be one if 3 girls in a class of 15 and you know all the boys have no boundaries and some were excluded from their last school for sexualised behaviour.
I'm not saying it's ok To impact a mainstream class. But it's not just parents being awkward either. In every case.