And to re-interate the original point - should transmen use the ladies? Where trans women go may not be women's problem to fix as they are biological men, but where transmen go is - they are biological women.
The main issue is where males must not go. And that is into female-only spaces.
That's because males who go into female-only spaces pose a risk not to themselves but to the females in those spaces. If females go into male-only spaces, they only put themselves at risk.
The issue is also not up to women to fix, but the law, which has already done so. So, here it is:
Legally, at least here in the UK, it is not allowed to exclude a female who identifies as trans from female-only spaces. That would be unlawful discrimination on the basis of sex.
It is also not allowed to force a female who identifies as trans to use female-only spaces, that would again be unlawful discrimination, but this time on the basis of gender reassignment.
However, to exclude a female who identifies as trans from male-only spaces is lawful discrimination on the basis of sex. Because males have a right to the same single-sex spaces as women where privacy is an issue.
In order to ensure that a female who identifies as trans is not unlawfully discriminated against on the basis of either sex or gender reassignment, the law in question - the Equality Act 2010 - states that alternatives should be offered instead.
(As you can see, third spaces is not a solution first proposed by women's rights groups and campaigners but one advocated for in UK equality law.)
P.S. The legal situation changes slightly if a female who identifies as trans is in possession of a Gender Recognition Certificate as under the Gender Recognition Act 2004, such an individual is now biologically female but legally male.
How a female who is legally male is treated then depends on whether a single-sex provision decides access based on biological sex or legal sex.
What this means in practice, where alternatives are not always available, is that females who identify as trans retain the legal right to use a female-only space. And while they can legally be excluded from a male-only space, it is unlikely that they will because they do not pose a risk to males in male-only spaces. In contrast to males in female-only spaces.
Thus, it is their choice which space to use, limited only by the lack of freely available alternatives.
Typically, many females who identify as trans and who are medically transitioning are very aware of when they make other females uncomfortable and will then seek to chose an alternative if one is available and if not, may decide to use a male-only space. At their own risk.
I hope that this answers your question.