Yes, the first time it was factual... Then he learned how much power to express strong feelings, how much attention, that such comments got him. In effect the reaction to that incident taught him that brown is different, brown is a bad thing and that's a way of expressing his own anger.
He isn't racist, if he'd lashed out at a kid with red hair or specs and had a right bollocking over it, he'd have had that particular difference highlighted to him and would have used that instead.
Being autistic means you struggle to understand, manage and contain or express your own feelings, asking him to understand or empathise with someone elses feelings is likely a wholly alien concept to him.
So you'll need to ask him not to use those words for some other reason, one he can understand as a rule, for a motivator he does grasp properly.
Give him an alternative outlet for when he is angry or frustrated, give him words he can use, a code if you like so that you know how he feels and he knows you know.
Life as an autistic person is intensely frustrating and baffling, and that's for an adult. For a child who doesn't yet know that most people's brains work differently, and all this weird shit makes total sense to everyone else but not them... It must be so draining!