Apart from, if you try to define a set of biological criteria you will fail. There are exceptions to every rule and people who don't fit into near categories.
Oh absolutely, I thought that went without saying as nearly any physical category will have edge cases.
If everyone was just seen as a person regardless of what was in their pants, would it add to peace and freedom because we wouldn't be socialised into restrictive roles, or would are those roles hard wired and we'd be putting the fox amongst the chickens?
Well, being male/female entails SO much more than "what is in your pants" - risks (by which I mean probabilities, not dangerous risks per se) are wildly different for things like strength, pregnancy, menstruating, loads of health conditions, violent crimes - so those would still come into play. Hence single-sex spaces.
But when it comes to using that to assume - and expect - what an individual person is like, what their skills are, what they want in life, what their character is... then yes I think sex should be irrelevant.
In other words, where sex matters, be honest about it and let's not hide the differences it can bring. Where it doesn't, treat people equally.