I am strongly agsinst the idea that trans people should be ejected from the military or public life (or other jobs, or housing, etc, etc). This is actual transphobia. However, they should expect to be treated as the sex they are, in these roles/in life generally.
However - men who
a) Appropriate female-ness
and/or
b) Get a sexual kick out of pretending to be (their idea of) female
and
c) Want to force their way into female-only spaces
... are by definition abusive.