IANAL.
I'd love a test case about single sex toilets/changing rooms and indirect discrimination if men claiming to be women are excluded from the women's facilities.
Toilets and changing rooms are about people's actual bodies, not their feelings about them.
It's all very similar to the prisons discussion. I know the JR of men in women's prisons failed, but that was because it was a Government policy and the EA doesn't say is framed permissively rather than prescriptively about single-sex provision. Also in that judgement, heavy hints were dropped (on my reading of it) that any legal action based on the entirely foreseeable harm to women which resulted from the policy would probably be successful.
I am assuming the courts would have to start from the principle that males with the PC of Gender Reassignment can't be less favourably treated than other males. I think the claim could fail right there because if there are male toilets with urinals and some cubicles (a tiny minority might not be able to use urinals) then there is no disadvantage. They can use those as easily as any other male.
They might argue that they felt unsafe in male facilities, but in a court of law, I would think you'd have to provide evidence that you were unsafe - statistics on assaults etc, not vague feelings. (But gay and disabled or SEN are potentially vulnerable, too.) Or they might allege it was an affront to dignity (why?) or privacy (but there are cubicles). And even if solid evidence was provided (unlikely) and accepted, that would not = may use women's facilities, because the discrimination would have to be relative to others of the same sex, who are also excluded from the women's facilities. It would logically mean third spaces where possible.
And since even males with a GRC can be lawfully excluded, I can't see how males without even that fig leaf could win, even if it took an appeal court to give the final judgement.