Out of actual curiosity, if a person had taken puberty blockers early enough not to go through male puberty, would you be comfortable with that person swimming in the women's races?
I think you are fudging the issue here. This is also like bringing DSD people into the argument. It's transwomen we are talking about, and mostly it seems transwomen weren't elite athletes as the biological males they were, so they think they will have a go as women.
Personally I would like to see puberty blockers banned, but that's a different argument for another thread.