You're not in the wrong OP. If someone is technically male, even if they think they are a woman (or grew up thinking they were, which can happen with some DSDs) and has the advantage conferred by male puberty, it is simply unfair for them to be in the women's category. That's why we have women's categories - so that women can play sport and get to compete and win.
Same as weight categories and age categories. You don't have boxing that allows anyone of any weight or age to fight anyone else because then only one demographic could ever prevail.
What absolutely bizarre about this crossover from trans "allyship" to trying to defend male boxers in women's categories, is their arguments are contradictory. So someone was thought (incorrectly) to be female at birth, grew up being seen as female (possibly) and has female on their passport. So that makes them a woman? But a person who was thought (correctly) to be male at birth, grew up as a boy including male puberty, and has M on their passport, can be as much a woman as any other, just by saying so? Does what you are "assigned at birth" matter or not?
I really think the thought process in these people's heads is something along the lines of "oh no oh no, women are saying someone claiming to be a woman is really male/a man.... and when women do that it's bigoted and wrong and they must be punished and I have to take the other side to show I'm a good person!!! Quick think of/copy some specious arguments and berate any woman who thinks otherwise!"
In other words this whole movement has successfully convinced a whole ton of ostensibly well-meaning, do-gooder type (often "lefty") people that caring that female people exist and matter as a category is wrong and any woman who thinks that way must be ostracised. And for many men, that unleashes a deep, often repressed feeling of superiority and resentment towards women that they now feel justified in indulging. I think a lot of people don't have a full awareness of those motivations though. They just jostle to be on the "righteous" side and feel good about themselves without thinking it through.
Sorry it's so awful for you and it's damaged a friendship like this. I think that's happened to a lot of people with friends, family members, spouses etc. But understanding the reality of sex is not morally wrong and I am convinced this mad moment in history will pass. It's taking its time but it will. Reality has a way of asserting itself in the end.