Ah no, it is exactly how it works.
Most people can perceive the natal sex of most other people most of the time. I'm sorry, but that is just how it is. I'm not one of these posters who thinks "you can always tell", but I do think enough people can tell enough of the time that for most people their sex is a significant factor in how they are treated all the time, and for everyone sex has been a significant factor at some times.
You are right about gender being "the psychological processes / social cues by which we recognise and assign identities to ourselves and others", but you have a simplistic view of this whereby the individual's self image is somehow able to control of the perceptions of others, and also is not in turn infleunced by others.
In fact, it's a complex, ongoing and crucially self-affecting process that starts as soon as we are born and never stops. By the time a trans women considers that he may be a "woman", his personality has already been subject to many many interactions with people who know him to be male and therefore are shaped by the expectations of male gender constructs. Already, his experience of life and the understanding of self that has formed has taken a different path to that of a person born female. And of course he knows his own sex, and understands and is influenced by the messages his culture gives him about what it is to be that sex even if later in life he will consciously chose to reject that construct and adopt what he believes is a female social construct instead. So even if in later life he passes perfectly as a woman and therefore is subject to exactly the same external treatment, it is still a different experience and challenge, and leads to different needs and outcomes, as when the same things happen to someone born female whose personality was formed under the knowledge of being female-bodied.
None of this is to say that people need to accept, embrace and live within the gender constructs society imposes on their sex of course, but saying that sex is not relevent to our social experiences and challenges and therefore no sex-based provisions are justified is just wrong, and it disadvantages female people in particular when genderists push this line.