We have single-sex toilets for privacy and dignity of both women and men. There's no reason for men to use women's facilities.
Aside from their everyday use, women's toilets have an emergency function that most people don't think about much. They are effectively ubiquitous mini-shelters.
Women have to navigate a public space where 50% of the population are potential predators.
Not many men are dangerous, but a few are. Women's toilets have this function of being an escape route - a public space where men cannot enter without immediately outing themselves as having bad intentions. They give women a potential chance to separate themselves from a man without alerting him and ask for help from another woman. (Single-occupancy toilets are less suitable for this purpose - but at least with mobile phones there's the possibility to make a call. As long as they're private enough.)
This function is particularly important in places where women are likely to interact with unfamiliar men.
Withdrawing the social convention that males just cannot follow a woman into the toilet means eliminating the majority of public area escape routes for women from men. This should not be underestimated.
(And as to the question, yes I doubt myself all the time. But I also doubt other people. If I am wrong, then people need to make more convincing arguments to sway me. I'm not 100% certain what the best overall policy outcomes are, but I know the current situation is totally unacceptable for women.)