You're definitely not unreasonable OP.
My DSis has severe learning difficulties and I saw first hand what my parents went through, trying to find a future for all of them.
My DM ended up with quite severe depression as a result and it very nearly broke her completely. I remember hearing her sobbing in bed one particular day, the thought of which still upsets me now, over 20 years later.
After much stress and argument, at 16 they found her a place at a residential SEN college and at 18 she moved to a permanent care home where she's been ever since (she's in her 40s now). The home is set up to care for a variety of levels of disability with some units available for those that can be more independent through to specialist units designed for those with the most complex needs.
My DPs learned that they couldn't be "nice" with Social Services. Everyone around them was being given places and assistance but my parents weren't getting anything. It seems SS were helping those that demanded things the loudest, whereas my parents were being polite. My DM's depression actually ended up helping things along as her wonderful GP got involved (those were the days!).
So, no OP, you're not unreasonable. I echo PP's point about trying to seek some supported living for your son.
My DM was the same as you - couldn't work after my sister was born as her needs were just too great, and there was no wrap around childcare for children with her level of disability. Once my sister had left home she did get back in to work. At that point she was in her late 40s but hadn't worked since she was 20. She got in to an admin job with the local council and worked for another 20 years until she retired recently.
She admitted to me over Christmas that as much as they love my sister, she did change their lives completely and that life would have been a lot more comfortable than it has been.
It's tough, and I send you many un-mumsnetty hugs!