I actually think there's some truth in this. In my, admittedly statistically insignificant and anecdotal sample of one.
I grew up in a 'middle class' house where my teacher/lecturer parents were far too intellectual and academic to give a shit about keeping the house clean and tidy (but too poor to afford a cleaner). Dusty piles of books everywhere, cat hair on the sofas, piles of clean laundry mixed with dirty in random washing baskets, chipped crockery, plastic bags hung on drawer handles used instead of a bin. It was a similar story at their similarly lofty and intellectual friends' houses. Also, I noticed that my parents and their friends might shell out quite a lot of money on something nice, like a nice car, or big TV, or a nice sofa, and then just let it go to shit. They wouldn't look after their nice things.
They had this sort of attitude that caring about things looking nice was an amusing personality quirk that other people have.
But when I used to go to my mates' houses who lived in the local estate near my school, they were always immaculate. Small, perhaps, but always clean and tidy. I used to wish I lived in a house like theirs instead of a crumbling, draughty, Victorian shit hole. I never invited them back to mine.
As a teen it used to really piss me off. I moved out as soon as I could.
But, as a counter-argument to the OP, DH and I are now vair middle class ourselves and I make sure our house always looks nice.