Interesting thread. I've also been on the end of the PA "posh" observations due to having a "royal" name, talking proper, doing reasonably well at school and these days dressing relatively eccentrically (vintage Goth on the Stevie Nicks edge)((charity shops,car boots sales, eBay etc)).
Family aspirational but never got beyond slightly bettering ourselves from Nana who went into service at 14. No university high fliers, some college, alot of "creativity" and scant property ownership on my parents end.
It used to grind my gears because I've always been relatively "poor" but people have always assumed otherwise. Currently at my poorest and effectively isolated socially because going for a drink means me or my cats won't eat. And my cats must eat.
Which leads to the side eye and raised eyebrows and the odd comment of "but you seem as if you should have achieved more so what exactly is wrong with you?" (And thereby hangs a tale too long and unbelievable to relate).
These days I can shrug it off, or come back with an arch "Oh darling, I'm classless you know" leaving them pondering which pigeonhole to stuff me in.
But I'm in my 50s now and lack filters, however I have sympathy for the OP because it can rankle, and it did in my younger years. I always felt it suggested I had ideas above my station and thought myself superior, when this is just the me, product of my environment and trying to fit in the best I could. In these divisive times it's irritating when you are "othered" even in jest (allegedly). So OP I hope you can figure out a way to knock it on the head and enjoy your socialising without your alleged poshness being mentioned.