The OP asks the wrong question, of course.
The issue is gender ideology: an ideology that says any man who claims to be a woman should be treated, in all respects, as if he is a woman, which means having access to spaces and activities exclusively reserved for women.
That isn't a tiny minority of men - that's all men.
For some reason, the OP brings up the Scottish prison estate. But until the Scottish govt lost a recent court case, it was policy to admit trans-identifying men into women's prisons. Men like Katie Dolatowski, who sexually assaulted a 10-year old girl in the women's toilets. Imagine how frightening it would be as a woman in prison to share prison space with someone like that.
Sall Grover a few days ago wrote this about a trans-identifying rapist, Benjamin Charles Jones, who had been found dead: "This man claimed to be a woman, spent years in a women’s prison, the women inmates were terrified of him, we pushed as hard as we could to get him removed but the Victorian’s human rights commission & politicians did nothing, and when he was released he attacked another woman."
The idea that we somehow shouldn't care about the dignity and safety of women just because it's only a "few" men is never going to fly. Imagine if we allowed adults to identify as children, so that a 25-year old could, if they wanted, join the Brownies. We wouldn't allow it, even if it was just one person. Even if they really believed they were a child, and they really really wanted to join the Brownies.
So, no, forget it, OP, with your disingenuous arguments. We are never going to accept an anti-scientific, misogynistic ideology that says men can be women, however much you pretend that it's "only" a minority and that they don't do any harm. We're not that stupid.