I'm 'old' (75), have a Freedom pass but don't use it. I have it 'just in case'. I'm sure you are right that many elderly do not need free bus passes but, honestly, you are opening a can of worms by suggesting they are not issued. Think of the outcry about the winter fuel allowance being restricted, despite many still getting it and not everyone needing it anyway.
I'm content with my lot. I paid my contributions while working (still do pay a little tax), and receive the full state pension. That's me, I've no complaints but there are elderly who are extremely hard up, struggling with energy costs and the general rise in the cost of living. They need the free bus pass.
We have to be sensible as we age. For example, many have big, draughty hosues. I had a semi with a garage, long drive, garden back and front. I sold it and bought a two bed flat where I am quite cosy, the bills are very reasonable and I can do as I please. I couldn't manage my house easily so it was a good move. I might move again to a similar place but with a little garden or private outside space, we'll see.
All I hear around me from similarly aged people are moans and groans about the government taking things away from 'us'; most of these moaners are very comfortably off, have good lives and go abroad sun seeking regularly. Fair enough, they are entitled to it, reaping the benefits of their working lives but why moan about not getting a few quid here and there when there are others who really need it? Children, disabled, others who are vulnerable, including some elderly.
I think taking free bus passes away from pensioners would be a drop in the ocean, frankly, would make very little difference to the economy. I don't believe in means testing, that stigmatises people. Either everyone gets it or none, same as free school meals. That way we ensure every child eats.
I doubt the bus passes will be taken away, they are essential for some but a lot, like mine, are unused.