Humans are dimorphic, whatever other categories we might want to layer on, this is the one which is clear.
It is clear in the sense that it's qualitative (dichotomous), but smaller framed people in their own sex are at a disadvantage too.
It can be argued that smaller framed men do not possess an advantage over larger frame/taller women.
The issue of transgender people in sports may well lead to a redefinition of how we separate sports categories, that's my point. How it will be achieved, I'm not sure. There will never be a truly fair way.