It's not about how often it happens. It's about the fact that there's nothing to stop it happening.
If you let trans women in, you have to let all other men in.
Both in a practical sense, because there is no way of distinguishing between a trans woman and a man (they are both just men), and legally, because the Supreme Court judgment has confirmed that if you are including trans women in a women only space you have incorrectly applied the single sex exemption and are illegally discriminating against other men.