Agreed 100%. It's rooted in emotional crippling of children, which is not limited to one sex but is applied and functions and results along sex-specific pathways because of that patriarchal power imbalance it always comes back to; and it's also rooted in the learned propensity to treat sex as a transaction that occurs outside the bounds of that power imbalance.
I was thinking much more directly, although to some extent it comes to the same thing. Fetishes seem to develop at leat in part through exposure to certain situations, particularly unusual sexual or sexualised ones, especially when the individual is at a particular developmental stage.
Maybe it makes sense that this is a capacity, because it means individuals can become sensitised as they enter adulthood to the sexual practices of their particular culture, which may be adaptive to the environment in some way. Because most of the time, in human evolution or even historically, what people would be exposed to would just be other pretty standard sex practices for that cultural group.
In our culture though, through a lot of wealth and leisure time, and the media, we've created a situation where young people are exposed to a lot of different sexual stimuli, at a very impressionable age, and often too they are not actually having sex until later on. So porn obviously, but even things in novels and regular tv and film, or advertising.
It's kind of an unusual situation historically. Even if we think of supposedly decadent societies, most people would not have been exposed to that kind of variety of images. Not to mention people often didn't have the privacy to spend much time on such things.