I experience this but in a different way. I have a child with Cystic Fibrosis. Due to medical advances people with CF generally live until their late 30/40's. The CF trust has been quick to put this about and many people have told me that it's one of the better conditions to have due to improvements. 
But again, like you, it's on a sliding scale. I know an 11 year old who passed away from it recently, due to the severity of her condition. My child is also quite badly affected and it really winds me up when people comment on how it will be fine for my child as they will live well into adulthood. They don't know how severe our child's CF is or how badly the lungs are damaged. My child has had quite a difficult and complicated time medically.
I hope my child lives for a long time yet, but I'm also acutely aware that my child is much further down the road than others. I'd prefer strangers to say nothing, than repeat crap stats and myths.
Likewise, I assume most things like DS and other such conditions have a sliding scale. I also think that the media has perpetrated such myths about DS due to anyone with DS in programmes being quite placid and gentle. Which fuels the stereotype that it isn't a difficult condition to suffer with, and that it is mild LD and a gentle loving nature. Im sure that people with DS have their own personalities and can be loving, happy, moody or sad, just as much as everyone else. It would be ridiculous to think otherwise.
I'm sorry that you are struggling too, do you have a community nurse that you are in regular contact with? Is it possible that she could provide you with the details of support networks and other parents experiencing the same type of difficulties as yourself. ((hugs)) for you.