That's your call.
I'm not asking about "passing". That's actually irrelevant when it comes to the question of how it can be justified that some men are "really" women (although your mind going to physical appearance is quite telling).
And to be crystal clear, the issue is not "a bloke in a frock" either. I'm all for blokes in frocks as long as they recognise they are still blokes!
The issue is a bloke who believes something in his mind makes him a woman (or makes him entitled to be treated as if he is a woman).
Frock or not, I would find that as offensive as turning up to a book club to find one of the blokes is wearing a T shirt with a sexist joke on.
It's belittling and disempowering. It puts women in the unenviable position of having to be "the one who makes a fuss about something silly" or to smile and pretend that being reduced to a chariacture is just fine.
It's an exercise in power. It's no different to all the times we smile weakly instead of making a fuss when a man makes a sexist demeaning joke at our expense, and he knows it.
Women's spaces aren't just about who sees us unclothed and whether we have suffered enough trauma to justify saying no to men. They are also about having conversations abour our own thoughts and experiences without men feeling entitled to reframe them for us, a space to experience being just us and how we are when men are not around, to help us see more clearly the subconscious patterns society lays out for us.