I think @Sunshineandchill is on to something with the 'bad actors spoilt it for true trans' trope, except I don't believe it's about individuals' choices, so much as an inexorable historical development that's played out over the last fifty years.
In the 70s, male transsexuals existed, on the strength of surgical techniques evolved during the previous fifty years, but they were rare, had no legal status following Corbett v Corbett, and had no protection from discrimination. So a political movement was born.
Medical practitioners of the time tried conversion therapies such as emesis and electric shock treatment, and discouraged transition if the patient was unlikely to pass, which kept numbers down.
Then it was decided that transition had therapeutic utility even for the non-passing. And a human rights based approach meant no more electric shocks (fair enough) and no more making a change in legal sex contingent on genital surgery.
So, for women, the situation has changed over time, from a rare encounter in the ladies (but not in other segregated spaces) with an almost-passing, virtually guaranteed castrated, individual with no political or social clout (therefore, likely to be frightened of the police, for instance). To frequent encounters, in every sort of space, with blatantly non-passing individuals who are virtually guaranteed to be fully intact. Since the turn of the century, lobbyists have been cheering them on, raising their social and political capital, boosting numbers, bullying women, and making any sort of sensible conversation impossible.
In hindsight, every step follows inevitably from the one before. So, what are the inevitable developments of the next twenty years? I don't think it's going to be good for women or trans people.