but as it stands the equality act allows for trans women except in rare cases to access those spaces too. That's just the law
It's not. The only males who identify as trans who would even be considered to have a legal right to ask for access to female-only provisions are those in possession of a GRC.
For all others, the comparator for discrimination claims are other males whose legal sex remains male. If it's legal to exclude a man who doesn't identify as trans, it's legal to exclude a man who does identify as trans as a blanket policy. Before excluding males whose legal sex is female however (ie GRC-holders), the provider must consider whether such exclusion is justified.
And under a total of six sex-based exemptions allowing for a great number of female-only legal set asides, the Eqality Act states that yes, even GRC-holders can be excluded from opposite sex facilities, services and provisions.
Furthermore, and just for your information, the Government Equality Office has confirmed in correspondence, that case-by-case justification for exclusion refers to the fact that each use case that is to be offered on a single-sex basis must be justified as legitimate and proportionate not each exclusion of an individual user.
So, to recap: the law allows for the exclusion of all males, regardless of identity, from all female-only legal set asides. Including GRC-holders. As a matter of course, and regularly as well as frequently, not just in rare cases.
Are you sure you're not mistaking policy for the law?