I think it's the blurring of the boundaries that worries me most.
I exchanged a couple of Tweets about trans issues in sport with a trans guy I know. The athlete in question wasn't Caster Semenya - who by most accounts is a woman with an intersex condition - but Andraya Yearwood, the trans girl who's been wiping the floor with other girls in US school athletics.
This person's take on it shocked me, as it was the first time I had ever heard the argument they used. They said that essentially there is in fact no such thing as 'biological sex', and that the so-called male and female binary does not in fact exist. Science is much more complicated than that and to say otherwise is to retain a primary-school understanding of biology. (And this person's mother is a biologist, and they themselves have a science degree.)
So when I said that Andraya Yearwood had an advantage, my friend replied that we can't actually say that, unless we were to test the biology of every athlete on the field (and who will ever do that, especially at schools level?). They said that hormones, muscle mass, etc. are all so variable in humans that you can't categorically state that one person (perceived to be male) is faster and stronger than another person (perceived to be female). The existence of intersex conditions illustrates this.
Things like Usain Bolt's 'twitch' muscles and a basketballer's height are natural advantages that all elite athletes have. Someone else has greater lung capacity. Someone else builds muscle faster... And we don't say that this is unfair. The next step in that argument is that we shouldn't say that Caster Semenya, with her naturally higher testosterone levels, has a natural advantage. And the next step is that, since we shouldn't say it about Semenya, we are transphobic and bigoted if we say it about athletes like Andraya Yearwood.
I was floored by this argument. I still don't know how I would argue against it, especially with a scientist. I also couldn't continue because this person has married one of my dearest friends (also a trans guy) and I fear that soon I'm going to lose their friendship over this. I don't know what to think about Semenya, but if this argument - which is fuelled by Semenya's circumstances - prevails in sport, then in ten years' time all elite athletes will have a certain thing about them that no-one will be allowed to say.