This is where it gets complicated Datun.
There isn't any law, as far as I know, around which toilets people use (although I believe there are H&S laws around the provision of single sex toilets in the workplace and possibly other legislation that affects schools?). Generally this is just a very strong social convention which self-ID is breaking down.
Refuges use an EA exception (Schedule 3 part 7) to offer services to women only, or separately to women and men, where this is a proportionate means of achieving a legitimate aim. Para 28 allows the exclusion of TW from women only services, again where it's a proportionate means of achieving a legitimate aim.
This is where the EHRC statutory code starts adding in extra hoops:
13.57
If a service provider provides single- or separate sex services for women and men, or provides services differently to women and men, they should treat transsexual people according to the gender role in which they present. However, the Act does permit the service provider to provide a different service or exclude a person from the service who is proposing to undergo, is undergoing or who has undergone gender reassignment. This will only be lawful where the exclusion is a proportionate means of achieving a legitimate.
13.58
The intention is to ensure that the transsexual person is treated in a way that best meets their needs. Service providers need to be aware that transsexual people may need access to services relating to their birth sex which are otherwise provided only to people of that sex. For example, a transsexual man may need access to breast screening or gynaecological services. In order to protect the privacy of all users, it is recommended that the service provider should discuss with any transsexual service users the best way to enable them to have access to the service.
13.59
Service providers should be aware that where a transsexual person is visually and for all practical purposes indistinguishable from a non-transsexual person of that gender, they should normally be treated according to their acquired gender, unless there are strong reasons to the contrary.
13.60
As stated at the beginning of this chapter, any exception to the prohibition of discrimination must be applied as restrictively as possible and the denial of a service to a transsexual person should only occur in exceptional circumstances. A service provider can have a policy on provision of the service to transsexual users but should apply this policy on a case-by-case basis in order to determine whether the exclusion of a transsexual person is proportionate in the individual circumstances. Service providers will need to balance the need of the transsexual person for the service and the detriment to them if they are denied access, against the needs of other service users and any detriment that may affect them if the transsexual person has access to the service. To do this will often require discussion with service users (maintaining confidentiality for the transsexual service user). Care should be taken in each case to avoid a decision based on ignorance or prejudice. Also, the provider will need to show that a less discriminatory way to achieve the objective was not available.
Julian Norman has written about which is the correct comparator to use in discrimination cases. It appears from her analysis that where a TW does not have a GRC, the correct comparator is a man. However if a TW has a GRC and is therefore legally of the female sex, the correct comparator would be a woman. There's almost no case law though so it's not clear how the law should apply in practice, especially if we're not allowed to ask for a GRC anyway.
Aside from discrimination, the EA also makes it illegal to harass someone because of a protected characteristic. No comparator is needed here. Harassment is engaging in unwanted conduct related to a relevant protected characteristic that has the purpose or effect of violating the person's dignity, or creating an intimidating, hostile, degrading, humiliating or offensive environment for them.
TW could claim it was harassment to make them use male facilities.
Women could claim it was harassment to make us accept male bodied people in what are supposed to be female only facilities.
We need case law!