I agree. Anything that involves dividing things, or people, into a 'this' and a 'that' inevitably excludes the thises from the thats, and as long as there is a sensible reason for doing it, it's OK.
'Discrimination' is not always wrong, e.g. not letting 12 year olds drive cars, or 65 year olds join the cub scouts.
And if there is a clear line between 'this' and 'that' there will always be 'edge cases' that are anecdotally and personally negative, and that's unfortunate, but it is inevitable.
When I was a child I missed out on a scholarship which would have changed my future prospects immensely, because my parents' income was just a smidgeon above the cut-off point. We were infuriated, and devastated, and I didn't get the educational opportunities I might have had, but I was an edge case, and the rules were there in black and white.
Segregation on the basis of sex has been proved to be justifiable in certain contexts, and there are only two sexes. That's it. 'A hard no', as you rightly say SlackJawedDisbeliefXY.