To go through the full gamut of arguments ...
No one can change sex
Therefore men can never have rights to women's single sex spaces
If they (men) feel unsafe in the men's, due to the presence of other men - why would that be women's problem to solve?
They can campaign for third spaces
What about other groups of vulnerable men? Why single out just this one group?
If they try the sex / gender bait and switch - gender identity is a metaphysical belief about the self. There is no objective, verifiable evidence that a man can have a 'female gender identity'. You can't force people to adhere to your metaphysical, faith based beliefs over their own scientific, verifiable ones.
Anyway, women's material realities (their sex) are vulnerable to men's material realities (their sex). Not 'gender'.
Basics of safeguarding is that you cannot tell well intentioned men from ill intentioned men. Exactly the same principle applies to trans-identified men. Women are not reasonable collateral damage because you've let the Isla Bryson's of the world into their spaces.
Women have rights too, y'know. They do not consent to men in their intimate spaces.
Any more?