This is the problem especially in the US.
But in almost every country there is a huge amount of social pressure put on women who speak out, and many find themselves the subject of death threats, and their families threatened, too.
Lia Thomas' team mates all said they were delighted to have him among them and didn't mind him sharing changing rooms - until one day the mother of one posted (anonymously) how intimidated her daughter and the other girls felt. Thomas had refused separate changing facilities and insisted on sharing with the women and "he is obviously attracted to women". (You can guess what that means.)
Once one had broken cover another couple stepped forward, and another and another.
At one race meet all of the girls held hands in solidarity and silence before taking their blocks. they have walked in silence around he pool. People are starting to support them - but every one of these girls knows she has now got a target on her back as a troublemaker.
Women in other sports (a skateboarder whose name I can't recall is one) are coming forward pointing out not only the unfairness to the girls competing in that particular event, but to the girl who loses her place on a team every time a man takes a place.
This is happening at non-elite level, too, and girls will drop out of sports if it continues.