"People of your class used to assert that about gays, as well. And people with differently coloured skin. And people who adhered to certain religions. Or people from certain countries. And so on..."
This is both a false characterisation of the aims of feminists who are campaigning to have sex prioiritised above gender identity when sex matters and these are false comparisons.
To exclude people who are same sex attracted is to illegitimately discriminating against those people for access to single sex provisions. The same in regards to religions or from different countries. All these acts would be illegal and considered immoral.
However, all male people are legitimately excluded from female single sex provisions. This is a form of legitimate discrimination that is based on safety and the privacy and dignity of female people.
Any person who rejects the same sex provision that an organisation has provided for people of that person's sex should not expect to then access those provisions that have been provided for the opposite sex. An organisation may choose to provide a unisex space that the person rejecting the provision for their sex but they do not have to.
A person's sex class is categorised as having a body that is formed around the production of either large or small gametes. It cannot change and it is not dependent on whether their body is producing those gametes at the time, has in the past or will in the future.
There is no person who is born in a body that is not their own.