When the embryo is developing into a foetus, it develops various primitive structures that will develop into the organs of the body. For a time it has primitive versions of both a womb, fallopian tubes and upper vagina and the vas deferens, seminal vesicle and prostate.
It also develops primitive gonads. If the SRY gene is present, those primitive gonads rapidly become testes. If it isn't, they become ovaries. At that point, eight weeks into foetal development, the Y chromosome has done its job and, save for being around to be copied into sperm, is now redundant.
The testes start producing testosterone, which leads to the abandonment and dismantling of the primitive female structures, the maturation of the male structures, the development of the primitive genitals as male rather than female, and, possibly, the masculinisation of the brain. Without high testosterone the primitive male structures will be abandoned, the female structures will be developed and the genitals will develop as female.
It is testosterone that masculinises, as is shown in various disorders/differences in sexual development such as congenital adrenal hyperplasia, and by trans men who take it to masculinise. While the default in some senses is female, all of us have the capability to masculinise on exposure to high levels of testosterone.
Fun Fact 1: Testosterone and oestrogen are chemically almost the same.
Fun Fact 2: The body converts testosterone into oestrogen
Fun Fact 3: Women have more testosterone than oestrogen. What makes the difference is that men have a lot more testosterone and a lot less oestrogen.