@OP
"How can I be racist toward you when I have no power?"
I think this is where you are going wrong. You can absolutely be racist towards people without power as racism is
Prejudice, discrimination, or antagonism directed against a person or people on the basis of their membership of a particular racial or ethnic group, typically one that is a minority or marginalized.
‘a programme to combat racism’
1.1The belief that different races possess distinct characteristics, abilities, or qualities, especially so as to distinguish them as inferior or superior to one another.
www.lexico.com/definition/racism
So if you you were to antagonise someone (shove them, yell at them in the street ect) simply because they are white you would still be being racist.
If you were to say 'all white people are X' or 'all white people do 'Y' (making a generalised assumption based on race) you would still be being racist.
However, although these things would still racist, this racism is not nearly as bad as the racism people of colour, particularly black people experience. That is the key difference.