I think it's linked to self composure and a particular type of confidence, as @Galiana said.
A person who appears 'classy' generally doesn't seem to be seeking the approval of the world. They appear comfortable with their place in it, but it's a calm, sort of arm's length confidence, rather than a loud/arrogant type of confidence.
So imo that tends to translate to not engaging in obvious attention-seeking behaviours, and also not looking like they are trying to fit in / be fashionable/ say the right thing.
I think it is statistically vastly more likely that you'll have that sort of quiet composure if your background/drmographics have supported your social- and self-confidence, so you aren't anxious about whether you are 'acceptable' /valid as a human in our society.
If you've lots of money, and have always felt like noone is going to question your right to be somewhere, then you're less likely to be anxious about that.
If like many of the rest of us, you've spent your life feeling like you don't or might not fit in, you're much more likely to feel anxious about that and be (consciously or unconsciously) seeking approval from those around you, rather than projecting an air of composure.