The thing is toilets already are gender neutral. No-one cares if a bloke walks into the mens in a dress, they really don't (apart from those men who would start a fight with anyone over anything), and no-one minds if a woman walks into the ladies in a tuxedo. Toilets are sex segregated, and while it was an honour system, it worked pretty well.
The trouble is people deciding that it's gender, not sex, and that they are special and should be given special treatment. (with a heavy dose of misogyny - men forgetting that women do have different needs, and we don't go into the ladies toilet because we might wear lipstick and a skirt, but because we want privacy from men to do whatever we need to do.