OP's point is one I keep making, so I like the question, even if the response so far has been a bit grumpy. (You meanies!)
You get TRAs going on about how they're nobly breaking down a form of discrimination by letting men into women's spaces, but they hope you don't notice that they're advocating for discrimination by "gender", and don't ask them why we should do that.
This whole thing took me about 5 minutes to reject about 6-7 years ago, by basically asking similar questions as OP to myself:
"Are there any circumstances where we currently use sex, where 'gender' would do a better job? Can I think of a single one? In what circumstances should I care about someone's gender and treat them differently because of it?" (I think I wasn't using the word "gender" at the time though - it would have been "sex they say they are", compared to "actual sex").
I couldn't think of any. If gender works better than sex, then that means it doesn't need to be sex-segregated, and can be mixed-sex. And once it's mixed-sex, you can't then justify separating "genders" because different-gendered people have no definable separate characteristics.
It makes as much sense as treating people with different star-signs differently. Society should be gender-blind, just as it's star-sign blind.
The segregation has to be by sex or not be there at all, because only sex is objective, immutable, binary and leading to different enough outcomes to justify the segregation/discrimination.
The same logic is quietly built into the EA2010 - discrimination by sex is generally prohibited, but there are a number exceptions, because sometimes it's important to treat sexes differently. Discrimination by gender reassignment is just prohibited, and there are no exceptions - it's never important to treat genders differently. (Correct me if I'm wrong). Although the GRA complicates that, as it complicates everything, once you get a "legal sex change".