in one sense elite sport is "unfair", because the vast majority of people will never be able to compete at those levels - though that's surely what we watch sport for? To see the incredible things that women and men can do?
And we recognise that there are certain things that make universal participation unfair, so the vast majority of competitive sports have categories to try to create a level-ish playing field between certain groups.
Competitors are split by sex (or used to be....). They are split by age (small age groups when young - Under 11, Under 12, Under 13....; bigger age groups when older - 35-39, 40-44, 45-49....). They are split by disability (blind athletes don't compete directly against deaf athletes, for example). And broadly this works - the rules get tweaked occasionally, but the broad principle is that it is necessary to ensure the safety and fairness of all competitors.
Yes, from time to time outstanding individual athletes emerge and put the others in the shade (Roger Black would surely have won more 400m gold medals if he hadn't been a contemporary of Michael Johnson). But that's just pure luck and there's nothing that anyone can do about it. It would be as futile as complaining about the rotten luck suffered by many promising athletes who didn't fulfil their potential because of things like the world wars, Vietnam, the apartheid era ban.
But we can do something about men wanting to compete in women's sports: we can say no. Separate events can be created for them if they really really don't want to compete in men's events - no-one is stopping them going off and setting this up.
Meanwhile they should stop fucking cheating and stealing away the opportunities that women and girls have worked so hard to achieve.