YANBU that it shouldn't be banned or automatically shouted down.
However, I think in discussing safety in changing rooms, showers, and similar, there should be more discussion on how these spaces are rarely designed with safety in mind. A wider conversation to have on safety in these spaces which yes should include risks of male violence among others that puts people at risk and puts people, particularly the more vulnerable, off from fully engaging in public life. The places we're most vulnerable in public needs more thought than is currently required.
It's a bit like people who live in areas of the least immigration are the most anti immigration.
And like the changing room debate, it's a little more complicated than that. Yes, significantly majority White British areas were more likely to vote Leave and are more likely to support stricter immigration laws, but it's mostly areas where this is true and there are wider social issues like high poverty, high unemployment, and have a lower proportion in highly skilled or managerial work. There are wider systemic issues involved than just exposure to other types of people and it's rarely as neat as media likes to portray. I live in an area with a lot of immigration and still have 3 elected ex-UKIP councillors that think harrassing hotels with refugees is part of their jobs...
Men have always been exposed to women, hasn't really meant they all see the discrimination, impact, or that our feelings are as our valid as theirs. It takes more than exposure - and that exposure might not lead to the same line of thinking.
There needs to be more consideration than just having it all be a free for all or about who people are in their hearts which is very individualistic idea that is terrible to put at a societal level. My relationships with trans people, including my trans ex, has made me more aware of the needs to really consider these spaces. I don't think those who just want to change the word on the door or open it all have actual vulnerable people in mind. It's just point scoring for them, it's not likely them whose been in a toilet or changing room and thought that something terrible could happen and no one would know for hours.