So you are saying if there is a additional need the person shouldn't miss out on life.
Well, no, but in practice it's trickier than that.
So I use a wheelchair following an accident ten years ago, I'm a bloody minded cow when I want to be, and I was determined to keep travelling.
Loads of places that say they are wheelchair friendly are not. I have crawled up and down stairs in St Petersburg's big art galleries because their website says completely accessible but they weren't. I've sat on planes for an hour after everyone else has got off and held up the next flight because "the lift isn't available, can you walk down the stairs?" No, sorry not unless you have god like powers.
It's even worse if it's autism or mh related in any way. I have supported a friend through a panic attack in a main line station because there were just too many people on the train and she locked herself into the loos for 40 minutes. We did eventually get in the train - about three hours after the one we were meant to catch.
Universities make reasonable adjustments for this kind of thing. They make the lectures available on video, you can be excused in person attendance (in one case my DD who uses a wheelchair wanted to attend but it was in an inaccessible room so she was told she was being excused).
You can also work remotely - no need to leave the house/do public transport etc.
Random other hobbies/socialising don't have to make adjustments and can often be full of rude people. (Why are you in a wheelchair? Etc).
So, yeah, people with additional needs should experience life. But it isn't as simple as just going out and doing it.