The main issue is a lot of people go in with the view "how can I not wear one" instead of "what can mean I could"
There should be comparably few disabilities that prevent you from wearing a mask, and these disabilities don't necessarily correlate with increased vulnerability. For instance having a nerve issue may prevent mask wearing but doesn't inherently make you more vulnerable, but someone with asthma might well feel confident in wearing a mask.
Both people should also ask themselves, what can I do to wear something, can I wear a light scarf, or a visor? If no, that's fine.
But as pp have said, many disabilities are unfortunately used to having to 'prove', rightly or wrongly that they are disabled. And these challenges are made worse by people claiming they are disabled when they are not. Truthfully, if more people are claiming they are disabled, people are less likely to be believe it and so be more probing in their questioning.
Other exemptions as well, again should be asking what they can do to wear one. Again if no, that's fine.
Its so so hard but there are an awful lot of people not wearing them, in all areas. If a proportionly small number of the population are truly exempt, then when you see, even anecdotally 50%, or whole families not wearing one, then you do start wondering. It's easier then to 'demonise' non-mask wearers and claim they're making everything worse and they should just stay home.
As normal the selfish few ruin it for those genuinely in need. Unfortunately it does mean more costs and responsibility on those truly exempt to prove it. Perhaps some claim back scheme, where you can pay, prove your entitlement and claim back the expenses of having to pay for doctors notes etc.
It's totally unfair but the world is full of selfish idiots who can't see past their own little bubble.