Things are not more complicated than that, really.
Yes, a person with internal testicles whose body does not react to testosterone is technically male.
However, since this person has no penis with which to use a pissoir, (or, you know, rape), and looks female, this is a non-issue except for this person and their spouse if/when they want to have children.
An intersex person with testicles but no penis will be AFAB - assigned female at birth, and in fact, intersex people without penis but with other male characteristics are the only ones who are assigned female at birth - actual women are recognized as female.
So, such a person, if identifying as female, is a lowly ciswoman according to patriarchal transideology!
Anyone born with a penis is considered male, and is awarded male privilege by males, and should therefore not have access to women's spaces.
I didn't invent it. Patriarchy invented the "penis=male" principle. As a non-bepenised person, I do not get male privilege, and I therefore want my private spaces to be free of penises.
If there was a "not able to get pregnant=male" principle and a woman could just have surgery and be awarded male privilege, then perhaps I'd see things differently.
But this is how males wanted things to be, and now they can fucking deal with it. Including those males who would like access to women's spaces.
Intersex males who have an unfair advantage when allowed to compete against women in professional sports are a problem, but a relatively small one because there is a limited number of intersex people - as opposed to the unlimited number of ordinary males who might "feel like a woman" if this feeling enabled them to win gold medals in women's sports despite them not even being good enough to compete against men at any professional level.