The answer to your question "Does someone else's gender really affect people so much" I would say both yes, and no.
I don't think most people consider individual trans people to be threatening, although the competitive female swimmer Riley Gaines might disagree, given her reported experience of having to repeatedly share a swim changing room with Lia Thomas, who had fully intact male genitals.
Individuals should be free to define their identities as they wish. However, alongside this, it’s crucial to maintain definitions based on biological sex —specifically 'male' and 'female'— in law and policy.
These definitions are essential to preserving protections, provisions and opportunities for women and girls, which have impact on daily experiences, safety, and fairness. This includes areas such as womens' prisons, rape crisis centres, sports categories, healthcare, and girls' education to name a few.