And this should also open the door to consideration of the fact that if you are arguing well obviously, no rapists or sex offenders but some men in some circumstances must be able to use women
then you are in fact accepting the rightness of men being provided with women to use as service animals and merely wanting to gatekeep a bit where that use breaks the women a bit too much. And looks bad in the press.
The issue is men using women at all. Whether as props or validation resources or furniture around their self identity, or as a toy to exploit and break. There aren't shades of it being ok. Again, unless you think females aren't equally human to males and have no business saying no to their birth duty to meet male need. In which case we're not talking about 'transphobia', we're actually talking about male supremacism.