Racism does not require intent which is why being colourblind is a confirmation that someone is certainly a racist.
If you are colourblind then you the fact that people of Black or Asian or other minority heritages underrepresented in the House of Commons or as CEOs or in other positions of authority or privilege is irrelevant because you don't see colour
The fact that all your friends are white is irrelevant, it just happened that way because well after all you don't see colour
The fact that all the TV shows your child watches, all the books you read them feature predominantly white people and where black people are featured it is often a mere token nod to diversity is not relevant as after all it is just coincidence, you are colourblind, you don't think about colour.
The fact your child only has white dolls, well it just worked out that way you didn't plan it. You are colour blind.
The fact you decided not to correct your child when he chooses to be unkind to another child because of his skin colour is fine because you and your child are colourblind and he didn't understand what he is saying.
Racism is not just using racial slurs to someone or physically or verbally assualting them because of their skin colour, the reason racism is so insiduous and painful is because it is mostly perpetuated by well meaning people who choose not to see it.
To be a non racist requires work, a lot of it, it requires thought, it requires seeing colour and doing something about the discrimination that people of BME heritages face. Being colour blind is comfortable especially for white people but it is not non racist.
As for the OP, why wouldn't you correct your child, what he says was unkind, if he said "I didn't share because he was stinky" wouldn't you tell him that wasn't a nice way to speak or because he was fat, had a disability, was a girl. Being unkind to someone because of something about them especially something intrinsic about them is always wrong. Why didn't you tell him that? It would be a first step for him in growing into a non racist adult.