Assuming they know these men are fairly decent types and they’re not at risk of sexual assault from any of them, some people seem to think if we’re not actually being raped we should happily let all men into female-only spaces but it’s not just physical safety it’s also about privacy and dignity.
Exactly, and to that I'd also add fear. If I'm in a female toilet or changing room or hospital ward and there's obviously a male there, of course there's a risk he'll actually attack/rape me, though that's probably not a high risk, and knowing that makes me anxious and uncomfortable - just like when someone's walking behind you at night and you feel relief when you realise it's a woman but are on your guard if it's a man.
The whole point of a women-only space is being able to relax and feel safe because no men are there - and as statistically transwomen have a higher incidence of sexual offending than men in general, it doesn't make me feel better at all if the man is IDing as a woman. He is statistically more likely to behave badly (and I don't just mean rape but things like harassment, masturbating, flashing etc).
It's like putting a big powerful barking dog in with cats and saying "he won't hurt anyone". Maybe not but the cats will still be on edge, stressed, unhappy, unable to relax.