That is the issue that gay people faced and still face in many areas of the world - that being gay is not "legitimate" as in a natural sexual orientation but it's viewed as a fetish or perversion instead.
The exact same thing is still happening to trans people.
Gender identity is a different kind of thing from sexual orientation, though.
Being gay is about someone's sexual attraction, so it makes sense that the gay person would know who they are attracted to. Whereas gender identity is about the person experiencing it saying "I know I don't look like it, but I actually belong in that other physical class of people over there".
A person being gay doesn't depend on anyone else agreeing with them; it's about their own sexual attraction (or lack thereof) to others. If trans rights merely meant that trans people should be able to wear what they want, have whatever surgeries they felt they needed and not be harassed or criticised for those choices, then that would be similar to the rights that gay people fought/fight for.
So when words like valid and legitimate are used.in defence of trans people, it is not said glibly but in direct opposition to statements like "transgenderism should be erased". Because while you may minimise the effect of such rhetoric and dismiss concerns about not viewing being trans as legitimate being dangerous to trans people, it's fairly easy to see how it would at the very least influence trans people to go back into the closet.
Obviously, trans people shouldn't be criticised for the fact they experience gender distress or gender incongruence, and in the majority of cases, I do believe that a trans person's experience of their inner sense of self is legitimate and that it exists. Aside from the inevitable subset of people claiming a gender identity for nefarious means, I do believe that some trans people genuinely believe they are the opposite sex. And that how they reached that belief is legitimate and perfectly understandable in many cases.
Many people experience deep distress related to gender oppression and their perceived place in the world, and they should obviously be supported in finding relief from that distress. Again, perfectly legitimate and they are in need of much support and understanding.
However, that distress or incongruence that trans people are experiencing around their gender doesn't actually make them the opposite sex, regardless of how fervently they feel it does. And it isn't reasonable to expect everyone else to ignore biological sex in scenarios where it matters to people, simply because some other people would rather not be confronted with their own biological sex.
Setting aside the obvious use of hyperbolic rhetoric designed to evoke an emotional as opposed to a rational response in people, if trans people genuinely do feel terrified that their very existence is being viewed as illegitimate, I suspect that is caused by a fundamental misunderstanding between how people who believe in an inner sense of gender identity view the experience of having an inner sense of gender versus how people who don't believe in gender identity see the same set of circumstances.
Personally, I see being transgender as more akin to being a Christian than to being gay - I don't believe lots of the things Christians believe to be true and centre their whole lives around, and I don't believe that their god as they see him exists and I don't see all things the way that they do, but I accept that they believe those things, that it's important to them and that they have every right to practise and express their faith. I don't agree with Christians when they tell me I'll go to hell if I do x or don't do y, but they typically know that they're free to believe what they want and I'm free to not believe, so we can all coexist quite peacefully and respectfully.
If trans people feel that they can only exist if everyone else can adjust the world and their own thinking so that it accords with how trans people would prefer the world to be, then obviously we are going to run into issues, because that's a wholly unreasonable expectation. If trans people insist that their world view is the only correct one and everyone else must live as if their perception of the world is true regardless of their own beliefs, and if not doing so means that trans people feel they won't be trans any more unless everyone else can do things their way, then I can see why they might see people saying no to them as threatening their right to exist. I still think the issue is a big misunderstanding, though.