I think he is probably more the classic public school boy who is wealthy, good looking, athletic and popular. Very charming to the people who are from the same background and interests, who he sees as equals and friends. The ones he wants to be associated with.
these boys can be extremely unpleasant and mocking to those who aren't so sporty, wealthy, good looking or popular. They grow up into men who form groups of braying hooray henries who move in packs and intimidate weaker, or what they consider "beta" people. Or you are just invisible to them.
there were many of these types at the public school I attended and I know it well. Think of Chuck Bass in Gossip Girl.
these boys when they turn into men can either marry a girl of a similar mentality or someone more sensible, like Kate. Then they either carry on being hooray Henry arseholes or become a more pleasant member of society. And they grow up themselves. Or a terrible tragedy occurs at a critical age and leave them emotionally "stuck" in the mentality of a teenage boy at Eton...
hence those people Harry considers "on a level" will get lovely, charming Harry. Other people he considers "sad", inferior etc won't find him quite so charming. Such as the matron at school. And the public he meets are of no consequence so he's just polite and charming as he has been brought up.