It's the usual thing.
-I wish to identify how I like and be whatever sex I choose!
-Fine. Lovely. There's a mixed sex facility over there where it doesn't matter, where you're entirely free to be yourself, and you don't have to be restricted to only a choice of your sex based provision. Have a lovely time.
-But I want to use the single sex provision because I like the feel of that best, even though I know it excludes women from their only available provision, and makes many others uncomfortable, unwelcome and subordinated to my (male) dominance of the resources.
-No, because that's unreasonable, selfish, unnecessary and frankly anti social. Other people have needs too.
-Waaaaaaaaah. <Heel drumming ensues.>
And it is only men. No women with gender identities are going to be harassing, dominating and driving men out of the men's single sex provision, a) because the men won't have it, and won't be expected to, b) they wouldn't dare, and c) however women might personally identify they don't behave like this to other people.