Unfortunately yes we do have to be more assertive.
I'm eligible for a disability bus pass, train pass and blue badge if I wished.
I'm currently housebound so obviously not using but when I am at points I'm out and about I have been challenged and questioned at times as none of my conditions (mental illness, spinal condition and muscular condition plus I have asthma that can flare in winter and make walking far difficult) are generally visible currently (although at points I've been on crutches but I'm actually being advised medically to avoid using mobility aids as much as possible so as to maintain as much muscle strength as I can).
I try and keep my temper and basically say things like
Because I'm disabled
Because I struggle with my mobility
But yes at times I get irritated especially if they're particularly rude and say
None of your business!
Yes I'm entitled to it and that's been decided by someone far more qualified than you.
At times it's been the people with me who've ranted back!
And they really let rip! My mother on more than one occasion has done the "you're very welcome to take her conditions on yourself!" Stance things like "She's in constant pain, constantly terrified, constantly worried about falling and getting a serious injury, her whole life is affected every day but hey clearly you'd LOVE to live like that"
She's a tiny wee grey haired Scots woman who appears very unassuming but not the person to get on wrong side of.
My aunt her sister also has a blue badge also invisible condition (chronic heart disease genetic condition but far worse now she's getting older) and she gives these types full pelt on that one too "constant pain, constantly fighting for breath, terrified she'll go into full failure, constantly exhausted - but sure you'd cope with that just fine!" 
Aunt is very quiet considered type but even she if alone (she rarely is these days as she's so unwell) will put them in their place. One time an older lady on a bus tried to make my aunt move from her priority seat "it's for us old folk not you fit healthy young 'uns just being lazy!" Aunt showed her disabled bus pass and used it to point to the "disabled" part on the "these seats are for elderly, infirm and disabled passengers" sign the ridiculous thing was there was another seat available this lady just wanted to have a dig!
There is plenty of info online AND members of the public are welcome to contact local council for info too.
@Berrylove and I'm not buying your faux naïveté any more than I think anyone else on this thread is. The op clearly explained why she has a BB and you are not entitled to her private confidential medical info anyway.
BB are not given out easily, so if you see someone has one you can reasonably assume there is good reason even IF they're bloody hopscotching into the shop or wherever!
A friend of mine her son has one as he has asd and appears very capable but he has no risk awareness, is easily startled into doing dangerous things like running into a road full of cars and is also hard of hearing so doesn't even hear car horns or a frantic mother calling him back.
She has become far more assertive with people like this too she struggled at first as she's quite a shy type but things like this plus everything else she has had to/is still battling in order to get her son the help and support he needs have led to her becoming more vocal.