Historically, men with young daughters, parents of children with SN, elderly gentleman with bladder problems, etc, could all use a female toilet and be accepted with compassion and understanding, as a courtesy.
No-one is disputing the odd exception to the rule, based on personal circumstances at the time.
This is only contentious, because of the recent bathroom debate with the 'come one, come all issue'.
When people are even questioning the OPs right to tell a man he is in the wrong toilets and she is answered with a 'fuck off, love', you have a mind shift which shouldn't be there.
Of course not all men are rapists, but all rapists are men. I never want to be in the position where even questioning the right of a man in my toilet is considered wrong.
And it won't just be one or two it will be any man, anyway who he knows he can. Bloody hell, if people can't see the problem with that they need to read the OP's post again. Aggressive male tells woman to fuck off for asking a legitimate question. (whether you are one of those who thinks she shouldn't have asked, the question WAS legitimate and the response was aggressive).