Okay, perhaps this sounds like playing with language and at the risk of going in circles, I see a distinction between racism (racial prejudice + the systemic economic, political and social "weight" to back it up) and racial prejudice (individuals not liking others because of their difference, but not having that "weight.")
In a similar way, you can have individual women who dislike men, even who might wish them harm, but you don't have the backing of history, culture, the law, traditions, social attitudes, etc. to translate that dislike on a large scale basis to harm huge swathes of men. However, when individual men dislike women and wish to do them harm, their position is at least to a degree supported by the history, culture, laws, etc. on a wider basis.
I think it IS important to look at whose views carry the most weight in society and where possible, distinguish between the attitudes and practices of people in those groups from the attitudes and practices of those outside the groups. Otherwise, there's the risk that everyone will be seen "as bad as each other" and the systemic oppression that underpins the bigotry of some will be ignored. Injustice will prevail.