I have some sympathy with this, and can see why it appeals.
In safeguarding there's a concept called gatekeeping. If you make very visible gates- procedures that demonstrate safeguarding is taken seriously- then predators go somewhere else. If you have no visible safeguarding mechanisms then predators are drawn to you as a soft target. So we do safeguarding 'big and visible'.
As soon as you say it's probably a really small problem, and we don't actually need to guard against this, then predatory people are attracted. Not the people you removed the gates for- they were perfectly nice old fashioned transsexuals who just wanted to pee in peace.
But when there are no gates, how do we keep predators away?
In safeguarding, everyone is treated equally. We don't safeguard against everyone except priests who are celibate and godly and therefore no threat. We don't safeguard against everyone except scout leaders, because they've been doing it for decades and everyone knows they are a good egg.
We don't assume teachers are ok.
Everyone is treated equally. No one is exempt from safeguarding. If women have rooms set aside for them, only women should be using them. No one is exempt.