That's exactly why we need to be talking about third spaces.
There are trans people, including non binary and otherwise gender non conforming people, who have said that they would like and would use third spaces.
And there are other trans people, almost exclusively trans women, who have said that they do not want third spaces because this would be "othering", and that if third spaces existed they would continue to use single sex spaces for the opposite sex regardless.
This conversation needs to take place in the public sphere.
Because I think almost everyone would agree that trans people need spaces where they can use the toilet and get changed in safety and comfort, and that they should not be forced to either use spaces for their own sex or be excluded from public life altogether.
But I think that almost as many people would agree that a trans person's right to safety and comfort should not come at the expense of the safety and comfort of the opposite sex. (Generally women, but some men also want and need single sex facilities.)
So if additional spaces are provided just for trans people, and some trans people refuse to use them, the debate will then move on to whether their motives are really legitimate. Because if a safe and comfortable space has been provided just for you and you don't want to use it, it's clearly not about safety and comfort. So what is it about?
Validation?
Because I think most people would agree that a trans woman's right to safety and comfort shouldn't trump women's right to safety and comfort, and even more people would agree that a trans woman's desire for validation shouldn't trump women's right to safety and comfort.
They need to say the quiet part out loud.