It's maybe worth pointing out, that it isn't just GC feminists and the far-right that worry about this issue. Mainstream conservatives worry about this issue too. They often don't think biological males in female sports is fair at all.
And it may actually be plenty of folks on the political left that would have concerns also. Remember that Twitter, the left-wing NGO's, and the political parties, don't necessarily accurately represent the general population, or the voters on the left.
So it's not like you just have this tiny fringe group of extreme feminists, and another fringe group of extreme right-wingers.
If you just take into account all the mainstream conservatives, or I don't know, polling of the general population, I think you can see that GC concerns about supposed "trans-rights" are perfectly mainstream.
But even if it was just two tiny groups of extremists that believed in this stuff, well you know, it's still going to be true that biological men aren't women.