Golden it comes down to biology again. Post-puberty men are bigger (on average), stronger, faster, more muscular. So if a woman plays with men on a reasonably equal footing, it's because she's supremely technically skilled, and her doing so is not putting them at any physical risk (their egos may get a bit bruised). If a man plays against women, he can be a bit crap and outplay much more skilled women, just because he is bigger and stronger - which isn't fair, and his greater strength can put the women at risk of physical injury which in sports like rugby, Aussie rules football, martial arts, and so on, is downright dangerous.
It's a false symmetry to say "woman playing against men" is equivalent to "man playing against women." Because biology.