The whole point of having a women's category is because of physical differences. Having watched school sports day (I have 2 to attend this week) the sexual dimorphism between the sexes is so flipping obvious. A Year 3 child runs a race and either a sporty-ish boy or girl can win. My year 8 daughter, who has been doing karate since she was 4 and does a level of running and cardio as part of that, has zero chance of winning against a sporty boy in her year. The adolescent changes have kicked in.
And while things are improving for women, we've never achieved true equality. In sports that has meant bigger prizes for men's sport. More TV coverage. Male athletes getting sponsorship (back to money). Women's sport has less prestige because we aren't the fastest or strongest in the world, not matter how hard athletes train. Or try.
It's not transphobic to try to protect women's sports. Sex is a protected characteristic, as is gender reassignment. This isn't a game of Equality Act 2010 Top Trumps, but it sure feels like it, and women don't have the top scoring cards.