I think society plays a part, too. If you live in a society where the only path is to get into a heterosexual marriage and reproduce but you can't, it's bound to exacerbate any problems.
Hitler obviously had many issues going on, plus the political and economic situation around him was terrible, and he latched on to the toxic antisemitism of the time.
Think about Caster Semenya. His sex was incorrectly observed at birth, initially raised as a girl, and at some point, he must have realised he looked and felt different. What was the impact? Well he had access to education and sports as a girl anyway. It seems that his community accepted him as he was. His disorder opened up amazing opportunities via sport. He is attracted to females and able to marry a woman without an issue. He is rich and famous. All good.
Imane Khelif - in a sexist, socially conservative society, his upbringing as a girl must have been very different to the boys'. Imane has made some money and fame but has been rumbled now. If he is attracted to females, what are his options in Algeria? He could never live or dress as Caster does.
Now think about the Imanes that didn't make it in sports. No money, no fame. Marriage will be a problem, sex might be difficult and conception as woman impossible.
So yeah I think how society treats people with disorders is crucial.