The problem is, it's asymmetric.
I am male, and I would not feel physically at risk if an average-looking female, however they might identify, were to come into the male bathrooms or changing rooms while I was using the toilet or getting dressed or whatever, because as a male of very average physical strength, I am statistically very unlikely to be physically at risk from an average female. It might be uncomfortable, but not threatening.
However, if I were an average female, I expect I might feel very threatened if an average-looking male came in to the dressing room while I was getting changed and we were the only two people there, regardless of how they identified, because I would know it was very likely that they would have the physical capability to overpower me and to rape me if they wanted to, with nobody else around to intervene if they decided to do that.
So even if that particular male might be a perfectly decent person, I would have no way of knowing that. The only information I would have about them would be their male appearance, plus the fact that they had decided to come into the female changing room or bathroom while I was alone in there and likely to take off at least some of my clothes. Given the high percentage of females who have experienced harassment or worse at the hands of males, I imagine that I would feel very threatened based just on that information. This perceived threat would have nothing to do with how that male person identified, since I could not possibly know that information, so logically my fear would not be transphobic in any way. "Male-phobic" perhaps, but the past behaviour of many males might make that rather justifiable.
@lateforschool this does not directly answer your question but maybe it will still be useful for your discussions with male friends about this topic?