I think it always gets derailed by toilets because that's the single sex space that you might visit the most frequently, but you actually are in it for the least amount of time. You also generally have full agency to leave anytime you like or feel uncomfortable so it feels safe.
Prisons, rape crisis shelters, hospital wards feel distant and unreal to the average person - oh most people will never experience that so it's not worth making policy for such extreme circumstances. But in fact, when you're there, you're almost certainly in a very vulnerable position, for a long time, and can't freely leave if there is an uncomfortable situation. That's exactly the kind of situation where a strict policy is needed to keep service users, patients and inmates safe.
I would also argue more adult women than men have been on a hospital ward overnight at some point, I was trying to describe to DH the vulnerability I would have felt having men on the ward at night after both my births and realised he has never been admitted to hospital in his life. Out of curiosity I asked a lot of friends and almost none of our male friends had ever stayed overnight in hospital (we are all mid-30s) whereas almost all the women had, and not just when pregnant or having children.