Yes, your interpretation is what I meant - I should have made it clearer.
You're right that the effect of any harm disproportionately affects females, but that is irrelevant to the aims of the gender-affirming movement and whose 'rights' it actually fights for. Simply put, it fights for the rights of people who identify as trans, male or female.
The gender-affirming view of the difference between men and women is very complicated. A lot of them refuse to define 'man' or 'woman' because they fear (a) using social stereotypes would invalidate trans-identifying people who don't fit them and (b) using sex would invalidate trans-identifying people as a whole. There's not much else you can use. They might say 'a man/woman is whoever identifies as such', but that's obviously circular reasoning. Sometimes, gender-affirming people thus believe there is no difference, or any difference is meaningless.
However, if a gender-affirming person were to consider this issue thoroughly, I think they would eventually come to my viewpoint, which is that a man describes someone who has the norms, roles, and expectations associated with males applied to him, and a woman is someone who has the norms etc. associated with females applied to her. In that way, my perspective on this specifically is on the gender-affirming side.
n.b. you put a question mark next to 'gender-affirming'. I'm just using it as the opposite of 'gender-critical'. It sees common use in medical contexts, e.g. 'gender-affirming hormone therapy', but I think those of the gender-affirming movement would rather frame it as the 'trans rights' movement, and most people here tend to refer to gender identity, identity politics, 'the gender brigade', etc.