*The issue is when a trans woman is referred to as a man at the level of gender, not sex. Saying a trans woman is male or was born male isn’t controversial — that’s literally why the term trans woman exists. It describes a change in gender, not a claim that sex was never male.
So no one is disputing factual statements about sex. The disagreement is about refusing to acknowledge gender at all, even when sex has already been clearly stated.*
In the first place, yes they are very much disputing factual statements about sex and claiming access to women's single sex spaces. Before that became mainstream, there was pretty much the live and let live situation you ask for. We knew transwomen were not actual women because actual women means female, but politely pretended and used pronouns etc to help them get along with their dysphoria. That's gone now because trans activists said transwomen ARE women and therefore should be in women's toilets, changing rooms, sports, prisons etc. So in order to keep single sex spaces, women had to clearly assert the truth that men and women means sex not identity.
Women cannot just let the trans movement co-opt the word woman to mean some kind of meaningless identity club because the word woman is all over our sex based rights. Swathes of law and medical science would need overwriting to not mention men/women and just mention male/female. You can't just rewrite history like that.
And there's really very little point because as soon as we did that transactivists would assert transwomen are female too. They're already doing it. Beth Upton, a doctor no less, claimed in court under oath to be a biological female. (He's male, a man.)