I think the reason is historical. Historically, women have been denied property rights that (some) men have enjoyed. Historically, it's been regarded as legitimate for (some) men to acquire property, or to aim to do so, and women have been treated as part of this property. Therefore, (some) men have been encouraged to feel entitled culturally, and sometimes legally, to 'collect' women as property - as wives, concubines, mistresses, prostitutes, etc.
This has created cultures where men could openly compare and contrast women's bodies in ways similar to ways in which people might examine and evaluate other forms of property (land, livestock, foodstuffs, clothing, etc.), and accordingly, the physical qualities of women have sometimes been seen to denote the status of those women's keepers (husband, pimp, punter, etc.).
Historically, women haven't enjoyed this 'privilege' in the same way that men have. Where women haven't enjoyed full property ownership rights, there hasn't been the opportunity for them to show off or compare their property in the same way (some) men could do. Nor have there been cultures and industries built up around women doing this in the same way that men who were wealthy enough could do (e.g. arts, pornography, related adornment, etc.).
And since, in these sorts of cultural and legal climates (i.e. where generally women didn't enjoy full property rights), many women provided for themselves by association with property owning men - (many of whom would have been no oil paintings!), it wouldn't have been practical or expedient for such women to express sexual interest in, say, the local farmhand, builders, etc., whose physical characteristics might have been far more attractive then those of the 'provider'. (Not that a load of male anxiety about the possibility of women's attraction to 'forbidden' males, usually from less economically privileged social groups hasn't existed. Consider white man's anxiety about the sexual prowess of African male slaves, etc.)
So, the idea that 'most' men are intrinsically attracted to women on the basis of particular physical traits developed, but the same presumption didn't apply regarding women's attraction to men - well maybe its possibility was acknowledged (see slave e.g. above), but it was a source of anxiety (or sometimes niche pornography), rather then generally and widely celebrated in the same way that 'the male gaze' and cultures, industries, philosophies, etc. based around the objectification of women were.... Women were seen simply to require a provider.
The idea's since been supported with reference to science (as posters have pointed out above) - e.g. the basis that men's reproductive capacity lasts longer than women's, therefore it's natural for older men to ogle younger women, but unnatural for older women to express sexual desire, that men are entitled to many female sexual partners, etc., etc....