Sex is not determined at conception. Most people think it is - but the only thing determined at conception is the plan for the development of a certain sex. For the sex to actually be developed, that plan needs to work out.
The gonads in human fetuses initially develop in a bi-potential state, meaning they can become either ovaries or testes. The SRY gene on the Y chromosome releases a protein called testis-determining factor (TDF). This protein then starts a chain reaction with SOX9 production (another protein), which causes the gonadal cells to form into the Sertoli and Leydig cells that make up the testes. If TDF is never produced or is interfered with, the gonad cells form into the Theca cells and follicles (which comprise the ovaries).
Once formed, the testes then begin producing a testosterone surge which typically starts in the eighth week of gestation and continues until the 24th week. This surge, combined with another hormone from the placenta, is responsible for the development of the penis and scrotum. Genitalia formation starts around week nine and becomes identifiable by the 11th week. If the surge does not occur, or the body does not respond to it (such as in the case of androgen insensitivity syndrome) then the genitalia form into the vulva, vagina, and uterus instead.
If there is an interference in this process then we can end up with the wrong bits, and this is the result of many intersex conditions. Oftentimes this is a partial development, where the external genitalia only partially form but functional gonads still exist. Sometimes children come out with fully functional male or female genitalia, but mismatched gonads. Sometimes the TDF protein fails to release and the fetus grows completely functional female reproductive organs, despite the presence of a Y chromosome.
The fact is, the vast majority of the population has never been tested for genetic karyotype, so we don’t know how common these cases actually are. Where does this come into effect for gender identity? Well, the exact same process that causes the external genitals to differentiate also occurs for the brain.
The prenatal brain doesn’t really start to develop until between week 12 and 24. The cerebral cortex, the thin outer layer of the brain that contains most of what we think of as consciousness, grows substantially during those periods of time. Prior to that, the structure present is more like a scaffolding — the basic parts of the nervous system necessary for bodily function. The primary sulci (the wrinkles in the cerebral cortex that allow for more surface area) start to form at week 14, well after the genitals have developed.
It has been confirmed multiple times via MRI studies that there are small but significant differences between cis male and cis female brains — differences which align with the gender identities of trans people in the study. Note that this does not mean that anyone with those differences will have that gender; gender identity isn’t that simple, but it provides evidence that there is a clear difference in masculine and feminine brains. There is also evidence that brains can have mosaic combinations of these differences, which may be the case in non-binary people.
A change in the testosterone levels in the fetus after the 11th week can directly impact the masculinization of the cerebral cortex, as well as changes in other parts of the brain structure. This has been examined over and over again in studies of female-assigned children with CAH (congenital adrenal hyperplasia) and CAIS (complete androgen insensitivity syndrome).
An excess of testosterone in the mother’s body during the second trimester can (and does) cause masculinization of the brain in an externally female fetus, and an interference in testosterone production or uptake can (and does) cause feminization of the brain in an externally male fetus. This interference does not have to be external in origin, either. Any number of genetic traits can cause the brain to respond differently to testosterone.
A fairly large study of transgender individuals released in 2018 found several key genes (https://academic.oup.com/jcem/article/104/2/390/5104458) which were statistically more likely to be longer among trans women (longer as in having more repeated fragments). Individually these genes may not have an impact strong enough to cause a malfunction of masculinization, but collectively they absolutely could reduce the ability for the fetal brain to masculinize. These genes are all passed from parent to child, giving credence to a tendency for trans parents to have trans children.