Only in gender identity circles and very conservative circles do 'woman' and 'man' have socially constructed sex stereotypes "attached". The rest of us recognise that 'feminine' men are just as much men as 'masculine' men, and 'masculine' women are just as much women as 'feminine' women. DW, for example, doesn't wear makeup (thank God) and isn't obsessed with removing body hair, and is a better plumber than I am. Those characteristics don't mean she is 'manly'.
Whether people recognise that socially constructed gender categories exist doesn't change that they do. A thing that even gender critical feminists acknowledge by virtue of being gender critical of trans people supposedly perpetuating sex stereotypes.
The personality traits and choices that are referred to as gender (as in conventionally feminine and masculine) are far more complex than a straight line from ultra masculine to ultra feminine, with everyone somewhere on that line. In any given society, some things are stereotypically associated with women and some with men, but they change over time, place, social category etc. For example, if you were to compare how men are expected to dress in a typical Church of England church (casual) with how men are expected to dress in a typical Mormon church in Utah (formal), you would see quite a difference.
But gendered associations still exist being the point.
And if 'non-binary' is purely a gender term, then many of us are non-binary. I certainly am, as I don't match the masculine stereotypes of UK cultures in every respect and have been known to have long hair (it was fashionable in my youth). I am also not as aggressive as men are 'supposed' to be in the UK; aggression was to some extent socialised out of me. So if a man squares up to me, I'm not spoiling for a physical fight; I have no desire to fight anyone. And yet, 'non-binary' is not clearly defined, and is not a word which is used to describe someone like me in the social circles I feel most at home in. In fact, I don't see any need at all for such a description, it just means normal people who are not slavish followers of cultural expectations for men and women. Similar to normal people who are not fashion victims.
Describing a phenomenon isn't an imposition of it as a self descriptor but I think its always helpful to have more clear language of it. If its a genderless society gender critical feminists are aiming for (one that I don't think is entirely possible given evolutionary pressures) then I don't see why this is so problematic other than for those who want to maintain associations to sex stereotypes that are entrenched in the words 'man' 'woman'.