It is a valid question. But
What if the male sexed person has undergone various hormonal treatments and surgery to make them as physically close to a female sexed person as possible? Someone who genuinely lives their life as female. Where should they go? What facilities should they use?
The elephant in this particular room is that the problem here is men. It's that transgender people feel too threatened with male violence to use male facilities. Ergo the problem is men.
This is not women's issue to fix. We are not support humans. We are not here to sacrifice our own dignity, privacy and safety to affirm the identities of those who are not female (and they're not). They are trans women; and yes, it is usually trans women, not trans men, who are making this particular noise. They have a different biology, therefore a different set of physical and safeguarding needs.
I would advocate for a third space. That's a space I'd also use in circumstances of needing to use facilities alongside my young son. I'd not use it if I were alone. And if I use the reasoning that I've been the victim of both male abuse and rape, I am 'weaponising my trauma' (which ranks amongst the most offensive phrases I've ever heard uttered).
'Third space' isn't good enough for the vociferous activists, however, and in this regard you have to ask yourself why not. If simply ensuring the privacy, safety and security of trans people isn't sufficient, exactly what is it the activists do want?
The answers are not necessarily comfortable.