Do you think that men who identify as trans are the only men who are vulnerable in men's toilets? Or do you think that there are other men who might be targeted in a similar way by racists, homophobes, xenophobes or violent extremists who hate certain groups of people?
This is important.
I agree with @BonfireLady that a man in a dress is going to be a target for some men. I've seen enough drunk male aggression to anyone who looks a bit different, or who they believe to be challenging them by catching their eye while not looking the same. As far as I can tell, the fury at anyone who steps out of line is a mixture of fury that someone isn't obeying the rules they think should be obeyed and a belief that men who reject (toxic) masculine norms are belittling and taking the piss out of those who do.
But.
As @OldCrone says, it's not just men in dresses. Be the wrong race, the wrong accent, the wrong body shape, the wrong clothes, laugh at the wrong time - an angry man doesn't need much excuse.
So why are we (society) even considering that trans women only might have a justifiable "need" to be allowed in the ladies? Simply because, and for no more solid reason than, unlike any other out group of men who risk and fear male aggression, this out group has laid claim to the name "woman" and thereby somehow convinced a lot of people that they have a right to female spaces that is not shared by other vulnerable men.
It's a con.
So trying to find some sort of "middle ground" for these "women" is to fall for a con. Because to even consider that "trans women" might have this claim is to allow that basic, fundamental lie to slip past unchallenged. That is why they want us to argue about the details toilets or third spaces or exactly where to draw the line - because it allows them to define the playing field as where the line needs to be, when the question should be why on earth there need to be a line anywhere other than the one that is already there between Man and Woman.