You answered your question in the OP.
For many men, often sexual viewpoints of women are entirely distinct from general relational/societal/holistic view points.
For many women, they are not. They are all part of one big picture that determines how they view a man.
A simple way to illustrate this:
Take a woman, make her beautiful, but then tell a man the worst things about her possible. She is horrible, she hurts puppies, she betrayed her family, etc. But she is really hot. Most men could still find her sexually attractive, and separate those things completely.
Take a handsome man, make him horrible, misogynistic, awful attitude to women, selfish, rude, cruel, criminal, sexist, perverted etc. But he is hot. Most women would not be able to still think of him sexually without being entirely turned off by all of the other bad parts of his personality. (most, not all)
Is the separation of the physical/visual and the "rest" that men have right or good?
Probably not, but it is the biology they have to deal with. Yes ideally they should control it. But controlling the way your brain naturally functions is hard for most men and women...