Obviously not everyone judges on looks, and there are many reasons someone might not like you that aren't looks-related, but I would say there are definitely some people who react very weirdly towards you if they think you're good-looking.
I'm no Cindy Crawford, but most people seem to think I'm pretty, I get a lot of compliments from nice people, which is nice, but here are some of the weird comments I've had:
"Wow, it's so great that you're friends with those girls, because women who look like you normally only hang out with other good-looking people. Yeah, I thought you'd be a real bitch and only want to speak to other people who are good-looking but you're actually nice." (Just so bizarre, I couldn't even respond...)
"Life must be so easy for you, cause you're beautiful." (Yeah, it was super-useful when my mum had terminal cancer, then died, then I got divorced)
From a man: "You know, you're really sexy and loads of people will want to fuck you, but no-one will ever love you." (Cheers, friend's brother who I only just met. No one will ever love you, because you're a wanker!)
And most people who are pretty, but nice, and who have had weird reactions because of their looks, don't talk about it because - as the OP has found - there's no way to bring this subject up without people thinking you're vain, arrogant, up your own arse, or whatever.
And besides, mean, judgemental people will always find superficial reasons to make mean, judgemental assumptions about others, so hey ho.