You’re not understanding.
The problem is people using relatives’ or stolen blue badges or using spaces without having any badge at all.
A disabled person pulling up in their car behind a car parked in a disabled space can’t see the badge displayed at the front.
If the occupant of the car gets out and is not visibly disabled, this means that they are more likely to not be entitled to use the space. It is obviously not conclusive proof.
Yes, many blue badge holders are not visibly disabled. However the vast majority of people who do not appear disabled are indeed not disabled. So it is perfectly fine to ask someone who does not appear disabled whether they have a blue badge, as long as it is done politely.
Believe me, the amount of “sorry mate I was just nipping in quickly, just let me move” responses he gets make it worthwhile.
Obviously the blue badge thieves/misusers will just say yes, and I am not suggesting that the questioner then check the badge in the manner of a traffic warden. But it might make them think twice about doing it again when they are challenged directly by someone who has a right to the space.