I have a brother (22 years old now) diagnosed with ASD at 15, I show many of the signs myself but was never formally diagnosed, I don't live at home with him, but my older brother does. I am very sympathetic because i have a clue about how difficult it can be to function with ASD but my older brother is always berating my younger one for using it as a get out of jail free card.
I always say - its a reason, not an excuse and there are some things that can be done and others that can't.
My ASD brother will never be able to function NT in social situations and will always be overwhelmed and panic in crowds. Depending on the person they will be hyper sensitive to things that NT people can dismiss and simply would be unable to deal with too much new input - but that sensitivity can vary by person. Common things are being distracted by little irrelevances because they stand out in your vision because you can't filter irrelevant things out. Being a supertaster and being very fussy about food as a result, being unable to deal with loud sounds. depending on how supportive you want to be you could try to find out what are his panic triggers and limitations and sensitivies are.
My oldest non ASD brother should know better as he has ADHD and dyslexia and so had had some of the same difficulties, but he ffels taht he was never given a pass with his reading and writing or his behavouir, so his younger brother shouldnt with his struggles, i think my older brother is blind to some of the accomadations we make for him and the ways in which he has limited himslef by not seeking the help he could. he pushes my younger brother and causes breakdowns and gets mad at my parents for molly coddling him all because he will not make any allowances.
On the other hand my parents do molly coddle my younger brother a bit, he has things that are childish that he never grew out of, small example but he would not pour milk in his own cereal bowl at the age of 20 as mum had always done that and he has resistance to change, my older brother made a big thing about this and many fights later my younger brother is now doing it himself and I use it as a reminder that sometimes he can do things he thinks he can't that the change will not be that bad just because he doesn't want it, and if he tries, if he pushes himself through the panic he can do it.
On the third hand he is never going to go to a concert even though he wants to as that would simply induce too much panic with the crowds.
TLDR version, find out what his limits are, try to push himon small things but try to have sympathy when he cant do it, its not the simple YABU/YANBU you are looking for, there is no one size solution to all of the things its going to take work to understand.
But you are in a position to opt out of these struggles which would not be unreasonable give he is not your problem as you do not live there but in that case YABU to comment/critisise on what he can an can't do if you aren't going to find out about his struggles. Its up to you.