It's not 'gender' binary, it's a SEX binary. And the fact is, we have female single-sex toilets and facilities because the male sex class on a whole, are dangerous. 98%-99% (depending on which country you get the stats from) of violence is from the male sex class. Domestic Violence, rape, will tell you how 'dangerous' men are. Ask rape survivors like me!
And, for privacy and safety, of course.
I also find it strange that society has flourish and reproduced for multi-millennia, had no problems discerning male from female, until 10-15 years ago, yet those like you are now saying we can't tell male from female. Funny that. Funny how it is only now, after multi-millennia, we 'can't tell'. How the fuck did our ancestors cope.
And yes, there is a there a difference between the male and female skeleton. Male skull and male jawbone, height, gait, lack of hips, limb length, hand and feet size, adams apple, square broad shoulders and finally voice. All of these 'tells' have been a part of society since the first male and first female were made. It's not a mystery how to tell the sexes apart. It is so fucking easy!