The prison population is a good indicator of violent criminality because the figures are reliable and they can be split by offence.
There are lots of crimes that are committed that don't result in a prison sentence, but it will be the same for every group.
If the claim is that the sample is too small, or trans people are more likely to face a prison sentence, or the proportion is incorrect, that is, there are more trans identifying men then we know, its interesting, but still doesnt warrant removing sex for safeguarding. These men are still committing sexual offences.
We could say the same about any group of men. There will be few men living in a small village, but it doesnt mean the statistics are so unreliable they can be admitted into the female estate.
The reason people are confused with your argument is that you acknowledged sex and have situations where you know its important, like sport, but dont acknowledged sex is a factor in other ways.
I have to remind you that to be protected under the PC of GR, a man just has to intend to transition, to obtain a GRC a man only has to have a diagnosis of GD. Neither of which changes anything about his body. Hes the same man as before.