Yes some people's sex can be difficult to discern, especially in the time it would take a security guard to take a quick look at someone to make sure they were going in the right toilet. There is ample evidence of non-feminine women being questioned about their sex in toilets and other gendered spaces, sometimes somewhat unpleasantly.
Ah, you're a toilet obsessive. Let's look at the bigger picture, shall we?
Do you think people are allocated to prisons according to their appearance, or that they should be?
Do you think which sports category people are allowed to compete in depends on their appearance, or that it should?
Do you think which hospital ward someone is treated in depends on their appearance, or that it should?
Do you think schools decide which changing room a child uses depends on their appearance, or that they should?
Appearance doesn't matter. Sex matters.