Oh dear, you've got this completely upside down.
We know that most males are not violent. But we also know that women and girls are at risk of male violence, which is why we keep all males out of women's spaces. We don't know which males are dangerous, therefore we keep them all out.
There is no reason why the same should not apply to males who identify as women.
There is no evidence to suggest that they are any less likely to be dangerous than other males. This is doubly the case with self ID, since literally anyone, even a rapist, can say they are a woman and we are supposed to just believe them.
Transgender males are male, like all other males, which means none of them should be in women's spaces.
They may feel vulnerable. They may feel marginalised. They may feel that they are female. Although they cannot possibly know that because they are not, and never have been, female. But none of these feelings are based in reality. The statistics show that over the last decade the number of trans people murdered has been less than one per year, compared to 2-3 women per week. They have the Labour Party, the Lib Dems, the Greens, half the Tories, the civil service, the police, the NHS, Stonewall and tons of other trans activist groups banging on about how marginalised they are, whilst other groups don't get a look in. They have charities and services dedicated to trans people, whereas women are denied single sex services.
As you say, FEELING vulnerable doesn't mean you ARE vulnerable.
The reality is that trans people are not particularly vulnerable, nor marginalised. They are in fact a very privileged group whose demands have been prioritised over the rights and safety of women and children.
Trans women are men. They have nothing in common with women that we do not all have in common with men. We share neither a sex nor any kind of identity, and there is no logic whatsoever for them to be in women's spaces or women's sports.
Literally the only reason for including them is that they want to be included. And of course, what they want matters, and what the rest of us want doesn't.