The men who lose out to other men under this system gain a lot by the way the system automatically puts them above women.
Even a man who is the dregs of society can make himself feel better by buying a woman for an hour to do what he likes with.
Or he might take his low position out on his spouse.
Even the wealthiest, most powerful woman probably won't be believed in court if she's raped, for example, especially if she knew the guy beforehand, or if he's her husband.
Not all men oppress women on an individual level, but that for every man in dire straits, there is a woman suffering just as much, with the added disadvantage of being a woman, and all the problems that come with that: productive responsiblities, primary carer for children/disabled/elderly, less chance of promotion, less chance of even getting a good job in the first place, losing out on pensions/pay because of pregnancy (workplaces can find pretexts not to hire a pregnant woman for example). That's before you get to gendered violence against women, lack of representation in politics (although they're happy to take our taxes)...