There are certain ethnicities in which people can be very convincing when presenting as the opposite sex. British males, generally speaking, do not. They may think they pass because no one's said anything, but typically people are just trying to be kind to someone they regard as vulnerable and disturbed.
Thing is, even this is generally a result of the people doing the looking and being convinced not being very familiar with people of that ethnicity. You'll generally find that other people of the same ethnicity have no problem at all correctly perceiving the person's sex. So the rule that humans are really very good at perceiving sex still applies.
There is no doubt a lack of insight/empathy at play for trans people. They don't see themselves as a threat, they only view the threat they perceive as coming from others. It's very very hard to see yourself as a potential aggressor and to understand why others might not welcome you in their space, because it conflicts with how you view yourself. Therefore, for trans people, women saying they cannot come into their spaces simply looks like yet more prejudice. They don't feel a sense of belonging with male spaces and male people and now the space where they would feel more comfortable is also hostile to them. I can see that there must be a sense of feeling that you have nowhere to go and to feel anger that you cannot access the female space.
I mean, bluntly? Too bad. It's not women's responsibility to manage other people's feelings for them, and we are absolutely entitled to resent the fact that the people in question are refusing to give our concerns any weight at all while at the same time insisting that we pretend they're indistinguishable from us. Trans people who are willing to concede that distinction we can potentially work with, but if they're not then we're being put in a position where we have to work against them. That's not our fault, and the situation won't be resolved until they stop putting us in that position.
Thanks to Snappity for coming along and demonstrating what I mean when I say that some people we can't work with. There's no way to create a workable compromise with someone whose basic position is "do what I say".