Ben Shapiro does agree with the feminist position on this: that male and female are biological realities and not identities.
This issue led me to Ben Shapiro and I've now listened to lots of his podcasts. I disagree with him politically on most things: he's a small government libertarian of a type we don't really have in the UK (his views on guns made me WTF!!) But he's rational, believes in free speech and debate. I like him, even though I disagree with him mostly.
He is not racist. his views on health care sound extreme to a UK audience, but many in the usa think the NHS is extreme scary socialism.
Tribalism, refusal to listen to views you disagree with, and the need to apply denegrating labels to people whose views you don't like, is the reason we are where we are politically and on the trans issue.
If you don't see that then the penny really hasn't dropped.