First of all, if the sign on the door just says "men" or "women", without specifying biological sex, then I should use the women's. Because that's what the sign says I should do, since I identify as a woman.
But it doesn't say "People who identify as women". It says woman = adult human female.
Having said this, I avoid gendered toilets whenever possible, because I feel like an unwanted intruder in either.
Questionable, but in one you definitely are and in the other the other members of your sex do in many instances feel threatened by one of them being feminine.
To address the specific question, I agree that a male toilet at work is unlikely to be physically dangerous for me unlike, say, a male toilet in a pub. But it would still be demeaning and embarrassing for me and any men I encountered there.
Why? And why more demeaning and embarrassing than when you enter a female toilet?
The big issue is urinals. Unlike in a women's toilet, you've got people standing there with their genitalia exposed. I think it's not appropriate for me to be in that situation while presenting as female. Cubicles aren't ideal but are a lot more private.
Quite. Hence why gender neutral toilets are a terrible idea. Either they have urinals or they don't, neither of which is good.
For avoidance of doubt, I simply can't use gendered changing rooms full stop. So I would need there to be a unisex alternative in order to be able to use, say, a gym or swimming pool.
Why?
But yes, no qualms with adding unisex toilets. The problem is turning female ones into them, while leaving male ones male.