I've thought about what would happen in the event you wanted a male ejected from a women's loo. I'm in Scotland, so I only really think in terms of Scots law.
If the staff of the facility were unwilling to intervene and the male is adamant on his right to use it, presumably you would call the police. As there is no law that prohibits people from using any bathroom they see fit, it is just social convention and people's willingness to follow this that means males typically use their own loo, women theirs'. So I think you would have to make a case to the police that the person involved is 'acting in a manner likely to cause fear, distress, or alarm' and hope that the police would view it as a public order offence.
However, the pendant in me thinks that provided the person in question was doing nothing more than using the loo for it's intended purpose, I don't see how the police could in any way view that as 'acting in a manner likely to cause fear, distress, or alarm'. I mean, they are using the facility for it's intended purpose. Going to the toilet is not a public order offence.
In short, I think a man absolutely can use a ladies loo, and there isn't a damned thing that can stop him provided he is genuinely using it for it's intended purpose.
I've used disabled loos in the past despite not having any obvious outward disability. I know it's not strictly the same thing, but if someone else challenged me on my use of a disabled loo, I would not, as a first recourse, immediately reveal my hidden disability to a total random stranger. I really do not need to 'prove' my entitlement to use the facility. If they believe I am not entitled, then they can call the police and make their argument as to why. Aside from the fact I am indeed perfectly entitled to use a disabled facility, I'd love to know what sort of argument they'd make to say that I'm not. After all, what can you possibly be doing wrong by using a toilet for it's intended purpose?