I do agree that on the face of it, it seems very unfair. The SC judgment I believe mentioned situations such as rape crisis counselling, and other such situations, as other posters have said.
However, I do feel that if you as a female are determined to be viewed , and believed, as a male, and have gone through as many administrative and even medical processes as you can to outwardly appear male, there will undoubtedly be social consequences. And, because we normally do not allow anyone under 18 to have these processes, women who choose to do this are adults, and must accept certain consequences as an adult.
My family member is one such woman, who now presents as male, and aside from the obvious " delusion " (my word for it) that she is now a man, also realizes that she probably will not be welcomed in female spaces, although she does not feel safe in male spaces. So she uses mixed-sex spaces or nothing at all.
It's very limiting, and she knows it, but she's an adult and made her choices. I worry for her, but it wasn't my decision to make.