The problem with the issue of who uses which toilet is that it is covered by more than one piece of legislation/regulation.
Well, the main problem is the recent rise of a tiny number of noisy and occasionally violent men making unreasonable demands, but apart from that... it's the mosaic of laws and regs that makes it so unclear.
The obscure para 61 in which the judge muses about whether or not it would really be discrimination against non-trans men if they were only allowed use the men's whereas TIMs could use the women's if the men's was completely equivalent to the women's in terms of location, quality, facilities, fancy soap - OK I made up the last one!
In that very specific circumstance, letting TIMs use the women's might not be discrimination against non-trans men under the EA2010. Note he only said 'might not'.
But the EA2010 is only one thing - what about the women who are now forced to share their women's toilet with males?
The judge has made clear in para 5 that
One clear consequence of the conclusion reached in For Women Scotland was that if, for example, a service provider provided a service to be used both by women and transsexual women, that service would not be a single-sex service.
So if the women were entitled to single-sex facilities under other rules, e.g. workplace regulations, or building regs., they could make the case that they were being discriminated against by failure to provide the single-sex toilets they were entitled to under a different set of regs.
It's not just equality legislation.
The judge also referred to things like 'propriety' and 'conventional decency' as factors in determining who uses which facilities.
edited to say cross posted with previous post which quotes M Foran, who seems to be saying the same as me about the tricky para 61. So I may be right!