Actually.... Lots of transmen compete in men's sports. I've known several personally who have, including contact sports.
Something which probably helps this stay quieter is that transmen who have medically transitioned for a few years, with or without lower surgery, are really hard to notice. Many transwomen, more so those who transition later in life, don't always quite achieve the same level of passing.
In the contact sports I played, transmen were welcomed. And I know of transwomen who play the same sport. Supported, and fully part of their teams.
Most times, transwomen don't win their races or sports. Many are actually appalling after hormone treatment. Wins can occur but not often enough for it to be some massive issue. And they regulate the testosterone level very closely, to the point that women who were born women and never identified out of that label, who critics have slated for not looking womanly enough, pretty enough, dainty enough, have been tested and discovered to have higher levels of testosterone despite being genetically female.
And there's a case I read about the other day where a woman has been prevented from continuing her sport because they discovered she has xxy chromosomes. As far as she was aware, she was born a woman. But people's paranoia about whether a woman is even a woman, based on something as superficial and insulting as her natural appearance, means she now loses out.
Testosterone levels in a transwoman have to be much lower than natal womens' natural testosterone levels, to be allowed to compete in women's categories. This is in part why many transwomen don't perform well at all.