Forgive me for not having the time to read the whole thread, so I do not know if my points have already been made.
I am an adoptive parent of 3 boys who had special needs and could well have been one of the children you describe. Unfortunately their greatest disability was in looking "normal" and hence other children and their parents were unaware of their difficulties. They were therefore seen as the disruptive children in the class, the ones that never got invited to parties or to play. Having also had other children without difficulties, I know the dramatic difference between how their education progressed, and also how they were treated differently by their classmates and their parents. Yes, they got invited to parties, even if they were the ones with the "mental brothers".
Behind the scenes of the other children/parents the school, medical services and ourselves were fighting so hard to get my sons' needs recognised, so they could get the help they needed. This wasn't at all easy - it took many years and eventually going to the highest level - but eventually their statement was amended to they could attend a school for children with moderate learning difficulties. That way the children they left behind in mainstream no longer had their education disrupted by them, and my children got the best possible education for them too. They were all in secondary school before "we won our case".
My children are now adults, but they still face the hurdle of looking "normal" but being unable to fully function in the adult world. I guess they are still judged.
I am now a foster parent. One child we foster was so disruptive/violent in school that the teachers/social workers admitted they were terrified of him. The whole class would have to be removed when he became angry - the other children and their parents were probably horrified. It wasn't fair to those children, of course - and equally there was little the school system could do. But our foster son has no particular special needs, it was merely due to the horrors of his past and being passed from carer to carer. Thankfully within a year of being with us, he had turned around - you would not know he was the same child and he is now able to take a positive role within the classroom. But once again other children and their parents would have no clue.
I am now going through the same thing with another foster child. I am sure we are judged. I am a bad parent, they are a bad child - I am sure other parents are saying "but they do not have special needs or a difficult home life" - and yes, it is unfair to their children. I am sure complaints are made, probably many...
I guess what I am trying to say is I understand, but as children do not come with signs saying "Tourettes", "ASD", "ADHD" or "Have been abused" around their necks, I am not sure I know what the answer is.