I would say that transmen with pseudo male genitalia should be in male spaces. They have chosen to transition and commit to that life so they should no longer be permitted in female spaces.
Transwomen may well, in theory, be permitted to do the same and have access to female spaces. Again if they fully transition.
What we see happening again and again is intact men self-identifying as women and waltzing into women's spaces and causing distress. The same is not happening to the same extent with regards to transmen in male spaces and so this is like comparing apples and oranges quite frankly (although I do not some cases of pre-op transmen taking the piss and entering gay saunas where gay men are rightly outraged).
On top of that we have to bear in mind that the statistics show that transwomen are the section of the population more likely to harass, intimidate and sexually assault women. Being that they are male (!) Transmen don't pose this risk to men in male spaces.
That is crucial to take into account. I am sick of transwomen and transmen being compared. It is not the same.
It bears repeating:
Transmen do not pose a risk to men in the way that transwomen pose a risk to women.
Transmen should be in male spaces if they are committing to be men, otherwise it makes a mockery of the point of transitioning.
Transwomen, could in theory have the same right but to women's spaces if the GRA wasn't made an absolute mockery of. We have to seriously tighten up the pathway to a GRC, revoke the certificate from those people who don't abide by the contract, and stop the tide of self-ID.
If these things can happen I'll entertain the question of allowing the vanishingly small minority of committed transwomen into female spaces. If we continue to head down the path of self-identification and affording men the classification "woman" when they've done fuck all but put on some makeup and heels, then no they can't access women's spaces.