I have also experienced violence and sexual assault from many men as I had the great misfortune to be a teenager in the 60s. When people say 'Not all men," my response is maybe not all, but an awful bloody lot of them. And what was overt in the 60s, 70s and 80s is simply covert now with a very pornified generation of younger men displaying an extra edge of hatred and spite as they realize we need them less and less and won't put up with their shit anymore.
However I think you are totally missing the point. As soon as some men are able to access single sex spaces because they are supposedly trans, then other men, trans or otherwise, will also feel entitled to access them too for reasons connected with trouser filth and general perving. And before we know where we are, we'll no longer have any single sex spaces.
You are right that men can assault us wherever they like, but why should we make it even easier for them? And in addition, single sex spaces are single sex simply because they are where we are at our most vulnerable because we are in a state of undress.
This is about far more than excluding trans identifying men - it's about excluding all men. And at a time when being trans means that men don't necessarily even need to look like a female and only have to feel a bit girly, while expecting to enjoy heterosexual relationships too, we are at far greater danger than we have ever been before.