Right. It's a fetishist fantasy, no more and no less. I actually did uses the men's loos once, as a younger woman when there was a queue in a pub. The only danger I felt was that I might slip on the pish on the floor.
A woman running in to a man's loo in desperation is in considerably less danger than a woman in a woman's toilet when a man who pretends to be a woman forces his way in. He is deliberately breaching women's boundaries and is already, by definition, a bad man.
The men I ran past in desperation to the sole cubicle as they stood at the urinal looked startled and a bit annoyed. If one of them had been a predator I might have been unlucky, of course, and I wouldn't recommend it, but I was really very attractive in my youth (the sort of feminine hot that men who pretend to be women literally never can achieve because they are men) and of course they weren't leaping on me to rape me in the bogs, they were just peeing.
But all men who pretend to be women and who push their way into women's toilets are, by definition, exhibiting predatory behaviour.
You might very well be in danger of being raped or otherwise assaulted by such a man. The absolute best case scenario is that the man pretending to be a woman is forcing you to take part in his fetishistic fantasies and display himself in front of you in his womanface .
Women have every reason to be fearful of men who pretend to be women to gain access to women's spaces. Men - even those pretending to be women - can just use the men's bogs like other men. They'll be fine, unless the men in there are gay, aggresive and hyper sexualised. Which would still be a man problem of course, and not one for women to solve by being body shields for them.