Feminism is about understanding and freeing males from oppressive and damaging gender constructs, too, and about realising it's "us" not "them."
I was taught about racism, sexism and feminism from a very young age and can't recall any aspect of "us and them" in it, ever.
An important point of teaching it early is that it takes practice and understanding to see how deeply embedded and systemic many aspects are, and to understand how far-reaching the effects of seemingly small things are. For example, when I was at primary school, my family had explained to me that research had shown that even teachers who believed they were not sexist actually gave more attention to boys, so I was able to ook out for this in the classroom and see where I might need to put my hand up more or read more or tell my parents if I felt I was missing out as a result.
My son understood at primary school that girls might also want to play football, so he'd invite them to play; he also understood that it was healthy for him to cry or show vulnerability if he felt it.
Realising how males and females are portrayed in tv and books and how this affects our beliefs about ourselves is another good thing to teach children very young.
I've known a few primary school teachers who think boys need more outdoor play than girls – and then reinforce this stereotype in their work with young children. Ideally, primary school teachers would be taught how wrong this is and prevented from such dangerous practices, but it also helps to teach children to be aware of these prejudices and how they're in danger of being shaped by them.
I see what you mean about seeing what's fair and unfair, but a lot of sexism and racism is subtle and built on insidious cultural influences, so we need the tools to perceive these early on.