That’s because we gender atheists, you big dafty!
There is no such thing as blue or pink souls. You can believe in souls if you like but we cannot arrange society around something that we cannot see, sex is material. To translate into language you might understand, Penis-people in one toilet, vagina -people in the other.
And no, amputating a penis doesn’t make a penis-person into a vagina-person, it makes them a penisless-penis-person (because humans cannot change sex and single sex facilities are divvied up on the basis of sex, the clue is in the name).