Gametes are not produced at birth, and that is not how gender is assigned at birth. Some trans people never produce any if they transition, and some people don't even produce any gametes whatsoever, yet they aren't viewed as being sexless.
You've completely misunderstood the relevance of gametes in defining sex. An embryo in its mother's womb develops as either a male or a female. What that means is that the body is organised around the production of either small gametes or large gametes. A body organised around producing small gametes (sperm) will have a penis and testicles and the other male organs. A body organised around producing large gametes (eggs) and being capable of gestation (having a baby) will have ovaries, oviduct, Fallopian tubes, womb, cervix, vagina and after puberty will have quite different breast tissue from a male human.
It's not relevant whether you actually do produce these gametes or not, or whether you ever go on to reproduce. What matters is that having the genes that make you male or female affects the rest of your body too, and determines what happens in your body at puberty.
Growing up male or female also affects how you are treated in all sorts of ways and what expectations other people have of you, and you have of yourself.
It's a moot point whether male violence is the result of having a male body or growing up male, or a mixture. For the safety of the rest of us, though, it doesn't matter why most of the violence and almost all sexual crimes come from males, not females - it just matters that we know it and we organise ourselves to try to keep the more vulnerable members of society safe from it.