You are stuck in the TRA's trap that the only two points are "Any man can be a woman any time" and "No man can be a woman ever". You think the reasonable position is somewhere between them where "Some men can be women sometimes".
But anywhere between the truth and a lie is still a lie.
This is not a pendulum settling on a midpoint. It is impossible to take even a tiny step away from woman as a biological sex without changing fundamentally what the word woman means for all women. And because sex is immutable and our sex does have consequences for our lives, changing the word that labels this, and with it the shape and rational for the women-only provisions that were only ever created in the first place to support our sex, hurts us.
The answer to this is to step out of the linear question TRAs want to constrain this to. Why can't we recognise sex as well as identity? Why does this thing that is not body sex have to take the same names and provisions as body sex, overwriting and negating it as if it's a new better version of the same thing when it's clearly and obviously something totally different, something that if it does exist (and I have no issue with society deciding it wants to reify personality classes as well as sex - we do that in an ad hoc way all the time through dports and politics and music and class itself) exists alongside and separate from sex.
Even your pendulum metaphor shows it. Swinging back and forth along its two dimensional arc. But take a step back and it's swinging in three dimensional space. It is the pendulum itself that creates the constraints it operates under.
Genuinely, take a step back, look outside the two points joined by a single line and ask why not have a book club, or a support group, or even political rights for people of any sex who feel more aligned to or comfortable with traditionally female-coded interests, presentations or ways of interacting? Why is this only possible to do if it has the label "woman"?
Or take a different step back, look from another angle - which you can't do on a line but can do infinitely in space - and you might wonder if there's a good reason a trans woman might benefit from or add to a woman-only book club, surely there are other men who would as well? Once you include a trans woman you've clearly decided sex doesn't matter so what's the criteria to include transwomen men and no other men?
Stop trying to shoe-horn "gender" into the pre-existing language, concepts, provisions and rights of sex. Give it its own name, its own identity as something that is not sex, and let us all explore what it could really mean for us free from the constraints of sex that came before.
Because one fixed thing we know in all these possibilities is that whereever gender goes, sex will still exist and will still have consequences for women.
So why on earth in all these possibilities would anyone want to take the one path that requires denying that truth and demands taking spaces and resources away from women for the benefit of men?