My friend went to the seaside in West Wales last summer with her two little kiddies. She has a blue badge as she has a chronic illness. Well, she got to the car park at 11am, and got the last disabled space. She took about 6 or 7 minutes to get sorted, as she has mobility problems and had to get the kids sorted...
And during that 6 or 7 minutes, FIVE different cars drove past - with the occupant(s) glaring at her. A couple of them asked her if she was leaving, and she said 'no' as she had only just got there, (and they looked pissed off.) One couple stayed there just gawping at her for a few minutes, in disgust, as she obviously wasn't genuine as no-one under middle-age can be disabled! 
Then when she walked off with her 2 kiddies and the backpacks (to the beach,) she looked back and saw this one man walk RIGHT over to her car to check there was a blue badge in it. She could see him rolling his eyes, and huffing and tutting. FFS! Why?
Utterly horrid.
And she gets stuff like this happening at least once a month. Because she is less than 40, and look 7-8 years older than she is... it is assumed (by some) that she is not disabled. 
And I hate to say it, but as a few others have said, the people who are rude are ALL 55 and over. No idea why.
I am also sorry @Calypso44 that you had to deal with this crap.