Yes, we do need to examine our attitude towards age that's exactly the point. I find it profoundly irritating when people are insulting and then claim, with breathless naïveté "oh gosh I didn't mean fat/ugly/old in a negative way". If you're associating age and gender with negative characteristics, such as bad driving, then you need to consider the prejudices you're revealing.
But you aren't just examining it, you are saying that just by making the comparison or observation, by noticing, it's negative and a problem, always. Your assumption is that the comment is the origin of the problem and somehow the attitude will change if people don't say such things.
Sure maybe kids "pick out any quality that seems different if they want to insult someone" but if that quality was race or sexual orientation you'd correct them PDQ. Why not with age or gender?
Yes, the problem here is the kids are using these things as a slur, be it race or age or something else. It's unkind, and also often rude in other ways, when kids do this to each other, and they shouldn't. They should be corrected for all of them. But it's not that race, or the age, or round head shapes, that are offensive. It's the context.
If on the other hand, you read a description of someone that says they "drive like a grandmother" or something similar, is that pejorative? Maybe, maybe not, the context might tell us. We know what it means though, someone who seems to drive carefully, excessively so, it's very evocative. It's fairly common among very elderly people because their reaction times are slowed down, maybe their vision is not as good, they become very cautious about driving, or even walking sometimes.
It's an observable human phenomena we have all seen, and a situation we might all find ourselves in eventually. If it's being said about a young person, perhaps the implication is that they don't need to be so careful, they are being a worry wart. But it doesn't say anything about whether it's stupid for elderly people to be careful about their driving.