I agree absolutely that 'racist' behaviour by black people to white people isn't the same as the reverse situation. A white person racially abusing a black person is an oppressive act that sits within a framework of oppression . The other isn't. It pretty much naturally follows that the parallel point applies wrt sex.
I'm uneasy about the practice of promoting a meaning for a specific word, such as 'racist', so that it only encompasses behaviour aligned with an oppressive framework. To suggest that white person stabbed to death by a black person because of his race isn't a 'racist' attack, because slavery etc. feels very wrong. This isn't a million miles away from mitigating or partially excusing the attack, and that can't be right.
Take also a situation where an Asian man has murdered a black man because of his race. Again it just feels incorrect to say this crime would not be 'racist' because it had not occurred against an background of oppression between the relevant groups.
It seems to me that where sex is concerned, the word 'misogyny' fits the bill fine for describing systemic abuse, and individual acts within that system. I'm slightly suspicious when people try to say acts based on a prejudice in the 'wrong' direction are 'not racist' or 'not sexist' are looking to minimise that behaviour. I can't see how the anti-racist or feminist movements need that additional proposition to stake their case, and I think some people could be alienated by it.
Buffy, I don't think, as a matter of ordinary English, that 'prejudice' is quite the right word for abusive behaviour of any kind. I've always understood prejudice to mean the state of mind that leads to that behaviour (literally, the 'pre-judgement' of the victim based on race, sex, etc).