I don't just mean toilets. Because legally there is nothing stopping men from entering women's toilets anyway. It's just societal pressure. And I'm not justifying anything.
I am laughing at the "we can always tell" brigade, but I am 100% certain you can't "always tell". That is a ridiculous thing to claim.
A discussion about it, without name calling, or making utterly ridiculous claims would be great. Find a way that everyone is supported. Because there are just as many trans men as there are trans women. And trans men need to be included in the conversation.