But sex isn't a social construct.
The analogy stands. You are privileged because you've never had to think about your gender identity. It matches your sex. No problem for you.
There are aspects of race discussions that are socially constructed even although race itself is a biological fact. See the Jesy Nelson black fishing scandal recently. It wasn't just the darkening of her skin that caused the issue. It was the associated behaviour, dress, hair, jewellery, sounds and so on.
Stop trying to simplify something complicated.