So gender = stereotypes?
In that case, I don't really understand why there is any debate at all. Being a man or woman is very clearly a matter of sex, not stereotype.
Gender-affirming people would presumably say that men and women are those who identify as such, who are associated by society with the stereotypes around one sex or the other.
There are two parts to this argument. Firstly, the notion that men and women are simply those who identify as such. I find this to be a very weak argument. I can't just decide that I identify as a cat or a dinosaur or an astronaut and therefore make that reality. "Identifying as" something doesn't really have any meaning, other than "wanting to play make-believe" about something.
The second part of your argument is that men and women are those who are associated by society with the stereotypes around one sex or another. But I don't think "society" considers people to be men or women simply because they adopt some of the stereotypes traditionally associated with the one sex or another. If you introduce a young child to a man dressed in women's clothes, they won't think it is a woman, they will likely ask why that man is dressed as a woman. If a woman chooses to have short hair and play football, society will typically see her as a woman who has short hair and likes football.
Yes, it is true that in recent times, society has forced many people to go along with the pretence of calling people men or women even when we all instinctively know that they are not, primarily because of the exhortation to "be kind". But most of us do actually know that trans people are not what they are claiming to be, as is often demonstrated by the accidental misgendering that is often committed by even the most loyal of trans advocates - occasionally, people forget, and their instinctive recognition of the biological reality of sex cuts through.