I ended up coming up with some stock phrases when I was younger, particularly pre-wheelchair when I'd use a crutch to get into the supermarket then use their scooter or lean on the trolley...
Can I see your badge:
'Yes, this is my badge, are you a traffic warden or police officer?... Then no you may not examine it.'
You're too young to use this space/you don't look old enough to have a badge:
'You look too old to be that ignorant/stupid/immature/rude...'
And if you don't fancy those...
'Please stop harrassing me or I will call the police...' and get your phone out to do so.
I had the same problem when I was younger, the general population in my town is on average, over 65, and it is not easy to get a blue badge unless you have PIP here, so theres a big group of people who for whatever reason (and probably pretty good ones) who feel they should have a badge, who do not have one and are pissed off about it.
It's shit we have to arm ourselves in advance with phrases, strap on our brave pants before we go out etc... but it is the way it is unfortunately.