In the case of refuges, I would have an entirely different opinion. For sports it feels a bit meh.
As a woman I wasn't good enough to be on the mens first team in my fav sport. I was good enough to be on the mens second team. We didn't have enough women to make a decent women's team, (and I would have felt uncomfortable about the fact that weaker male players weren't allowed on it).
So that is where I played, in the mens seconds. As long as I can play competitively at a plausible level to my skills I don't care what the team is called, or who I share it with.
This attitude would lead to the top few levels of any sport becoming dominated by XY individuals, but after that it would be a fairer free for all than currently exists.
So it is really a matter of role models. I have never been clear why I can't be inspired by someone with a cock, just because I don't have one. I also don't know what about someone like Jess Ennis-Hill coming back from pregnancy so amazingly is supposed to be transferable to my own physique that was shattered beyond the ability of medical science to repair by the process. Honestly I don't feel I have significantly less in common with Usain Bolt than with Jess. I can't ever be like either of them.