I don't agree with naturalising sexual violence as though it simply flows from chromosomes, hormones or genitals. Sexual violence is shaped by social, behavioural and situational factors, not reducible to anatomy.
Yes! You are sooooo close!
One of the most significant ways in which society shapes us is in how others, personally and culturally, react to us, frame us, place expectations on us, present role models and limitations for us, because of our physical sex. Not our own perception of ourselves but how others see us and treat us. We don't even need to know that is what is driving it to nevertheless suffer the consequences.
The body is the crucible of the mind. The mind experiences the world through the body, and what happens to the body, directly or through what we experience because of it, shapes the mind.
And even your putative perfect trans person, the one not in any way influenced by social factors in forming their transgender identity, but simply suffering some sort of colourblindness or phantom limb effect in their ability to perceive of their own sex, is not immune from this.
This is why even if you believe, as I do, that social factors shape the population differences we see between male and female choices and behaviours at least as must as any internal sex-based nature, it is still never going to be true that a trans identifying male person has the same needs and challenges, risks to consider, opportunities available and fenced off, understanding of self and of possibilities, than a person whose self was formed living in the world in a female body. The things women need, even when it's due to social factors not purely physical, we need because we are female.
I would love to hope, some day in the future, there is a world where male people are not socialised to treat female people as they currently are, and female people are not socialised to expect it and accept it. (Though sadly, I think the pornifiction of society is taking us the other way). But until we get there, we cannot pretend that our sex is not a factor in the way our lives proceed.