Society has changed. The golliwog has grown in repute, he now stands for something his originator coudn't have foreseen... after all, her patronising view was that he was "a horrid sight, the blackest gnome" - so not human, perhaps! But he turned out to be a really jolly, helpful chap.
If this was the originator's view, it was incredibly racist. A colonial, patronising kind of racism is still racism, and helps to build an atmosphere where other kinds of more obvious racism are condoned.
We have seen examples of this kind of effect, recently, where in the light of the US Republican administration's Muslim travel ban, racist people in the US have felt emboldened to carry out racist attacks.
Just because Upton didn't mean to be racist doesn't mean that she wasn't, or that it wasn't harmful.