I read a quote years ago which went:
At 20 we worry about what people think of us.
At 40, we don't care what people think of us.
At 60, we realise they haven't been thinking of us at all.
I'm not that fussed what people think. People in general, that is. My family, my friends - their opinion of me matters, because they matter to me. But the average person on the street, not really.
I don't take this to mean I can be rude or whatever, because I don't care what they think. I am polite, and (I hope), kind, because it makes everyone's life a lot more pleasant. But if someone makes an unpleasant snap judgement, I might think about it for a second or two, and then I move on and forget it. If I don't know them, their opinion mostly is of no relevance.
And as someone else said, I have come to realise that when I am doing things, I am mentally working through the checklist of what needs to be done, whe I need to be next, I'm not looking round and judging anyone else. So I tend to assume that no-one is really that interested in me. It's the opposite of being self important!