I don't see any hysteria over toilets, just (mainly) women who are being forced to defend the last line of privacy, dignity and safety of everyone.
As a 55 year old woman I have been on the receiving end of SA's verbally and physically in primary school, secondary school, at university, at work, in a family member's home, on the street, in pubs and clubs, on the beach, swimming pools, busses, trains etc and yes in toilets.
I've not complained about them at the time because......well what would be the point, the men in power were usually the ones I would have needed to complain to or there was a prevailing culture that when it comes down to he said/she said the men are believed, sometimes congratulated, and the women dismissed as over reacting, misunderstanding and yes being hysterical.
I'm sure if I took a straw poll of all the women I know that the same would be true for most of them and it is something that I passionately do not want for my daughter.
So the last line of safety, dignity etc is the toilets. Given that not all women have need- thankfully - of the other services that should remain single sex such as rape services and the female estate of the prison service the one common denominator is that we do need at times to use the toilet and if that line is crossed then women truly have no safe place in public and no voice.