We live in a racist culture, so yes, small children pick up on things and then because they have no filter they feed it back explicitly. They aren't being racist but they are telling you what they have learned/noticed. Adults (especially white) usually don't notice these things because we are used to them and take them for granted, or try to minimise out of politeness.
The best thing to do is keep pointing out/explaining the opposite, without worrying about it or making it into a big thing. For example you could say things like:
Somebody having brown or pink skin doesn't tell you whether they are clean or not. Being clean is about how much you wash your body. Everybody gets dirty, and we have to wash ourselves to get clean. People with brown skin and people with pink skin get dirty and wash to get clean again.
That person doesn't have brown skin because it is dirty, it's just a different colour. Just like X has brown hair, and Y has blonde hair. X's hair isn't dirty, it's just a different colour. It's brown even when it has just been washed. (If you have pets, you could also show this - animal skin is often coloured like the fur, so you can look at their paws or tummy or move the fur apart gently to look).
Point out black characters on TV or people in their class or teachers at their school etc who they like and say "X is black and they are a nice person." If you don't currently have much black representation in your books, TV etc consider seeking some out.
Somebody having brown or pink skin doesn't tell you if they are a good or a bad person. That's not something that you can tell about somebody from their skin colour. You should look at what people say or do. If they say and do nice, kind things then they are a good person. If they say or do mean things, they aren't a nice person. The colour of their skin doesn't make a difference.
At four, it's totally normal to be very black-and-white about things (pardon the pun) because they have to go through a stage of understanding binaries/opposites before they can see nuance and shades of grey. That's why everything is always extremes and rigid at four. People are good or bad, nice or mean, clean or dirty. They have to try and make sense of the world using this framework. You don't need to pretend that race and skin colour don't exist, because they aren't stupid and they do notice. But explaining that it is unrelated to attributes of the person themselves can help them not to use that as a marker.