And this is why you can't base rules on whether a trans person passes.
Firstly, because the number of trans people (let's be honest, trans identified males) who pass well - not perfectly, but well - is really tiny. Even with the lower standard of "passes at a glance in the loo", it's still a tiny minority.
There are a lot more who have talked themselves into believing they pass when they really don't. Honestly, I think their real fear about going into the gents is that they won't be challenged.
And there's a specific demographic who know they don't pass, but then transgressing women's boundaries is how they get their rocks off.
How do you set objective rules that distinguish between those groups?