There are so many differences.
Neither men nor women are a "minority group" in a broad societal context, as is usually the case with one or other ethnicity.
Men and women come from and can be found in every family, every region/country, every community and every culture across earth. Often, different racial groups are distinct from one another in terms of one or more of these categorisations. Which is why race can mean a lot more than just race, sometimes.
Race is subjective. Fundamentally, it is a social construct to define, describe, categorise (and in some cases exclude) people who "look similar" as according to arbitrary criteria set by a particular culture/society. For example, indigenous Australians, dark skinned South Indians and people from very different parts of Africa might all be termed "black" if viewed from a white European perspective. Realistically they have nothing particularly in common culturally, historically, genetically or otherwise other than dark skin eg "not whiteness".
Sex on the other hand is an objective biological reality for every single human being who has ever lived, and indeed all mammals.
They are both protected characteristics because they can lead to disadvantage. Not because they are both as biologically real as one another. Biologically speaking, sex is real and race is not.
When it comes to biology - women are smaller, weaker, more easily injured and killed, vulnerable to sexual assault and impregnation, and carry nearly all of the reproductive burden of our species. There are no such biological differences between different races.
When it comes to social disadvantage, within white European culture, black people have and continue to be disadvantaged. The historical (and in many cases current) context that exists, is of white colonialism, slavery and racism. So a black only group would potentially be a way of offsetting that disadvantage within that context, whereas a white only group is more likely to be the direct opposite, unless there is a very specific reason for it to exist. Women have and continue to be disadvantaged. Their historical (and in many instances current) context is also one of oppression, slavery and sexism.
Without women's only groups, women will be further disadvantaged, excluded and in some instances frankly put at risk of harm. But without white only groups, the same cannot be said for white people (I'm adding a disclaimer that there might be some specific context in which a white only group is justified, I just haven't thought of one).
Forgot to even mention that being of male sex is a risk factor for violence and particularly sexual violence in a way which that just doesn't really compare to any other characteristic. Just under 2 women are killed by men every week in the UK.