I first came across this expression during A level English around 35 years ago. It was in DH Lawrence's The Rainbow, I think.
It's something that's been expressed to me about myself quite a few times since either directly, the coarser version of ^you can't polish shit', indirectly where the sentiment has been communicated if the expression itself wasn't used or just being told to 'do better' appearance wise and failing.
It's how I've come to think of myself throughout the years. I stopped making any kind of effort with my appearance about 3 years ago. I never dress up, I don't wear make up. I just don't try to look nice anymore. Clean, yes; tidy, yes; presentable, yes but not nice or attractive. I feel quite uncomfortable at the thought of 'making an effort' nowadays.
I find I receive far fewer negative comments about my appearance since I stopped bothering at all (well, none actually). Obviously, I don't get positive ones either but they were few and far between anyway. I can live without those. I just got really sick of being told I should try harder, try differently, be better and never quite managing it anyway. Or just being mocked for trying but failing to meet other people's expectations.
Has anyone else just given up on trying to look nice and found that once they were no longer perceived to be doing it, no one cared at all?
I don't know what the aibu would be.
YABU - you should always try to look your best even if your best isn't great
YANBU - I stopped trying (or never did) and also noticed less negativity
(Obviously, the people commenting negatively were mostly my mum, men and a couple of female friends over the years).