I know this is meant to be lighthearted OP, and I kind of get what you mean, as I have phrases I don't like, but would never dare admit them on AIBU! 
However it is a good cover all. I say it. Though technically it's true for me as I can drive but I don't as I have no spacial awareness, can't tell my left from my right, and my coordination is terrible. And my concentration is also awful, and I have OCD and terrible anxiety!
I passed my test on the fifth attempt, with beta blockers! I did it because I felt I had too, and because my partner guilted me into
It. Where I am now is awful for busy roads, and to get anywhere would involve either a very steep hill and getting out onto a busy road, or another way which involves turning into a busy dual carriage way.
I am happy to walk, and get public transport, and if I am offered a lift, pay petrol, parking etc.
I do feel that non drivers are seen in a bad light, but trust me you don't want me on the roads!