I don't really think it's defensible anymore, to be honest.
From Enid Blyton...
Once the three bold golliwogs, Golly, Woggie, and Nigger, decided to go for a walk to Bumble-Bee Common. Golly wasn't quite ready so Woggie and Nigger said they would start off without him, and Golly would catch them up as soon as he could. So off went Woogie and Nigger, arm-in-arm, singing merrily their favourite song -- which, as you may guess, was Ten Little Nigger Boys.(p. 51)
Agatha Christie novel 10 Little Niggers
Ten Little Niggers is the name of a children's poem, sometimes set to music, which celebrates the deaths of ten Black children, one-by-one. The Three Golliwogs was reprinted as recently as 1968, and it still contained the above passage. Ten Little Niggers5 was also the name of a 1939 Agatha Christie novel, whose cover showed a Golliwog lynched, hanging from a noose.
And
The Golliwog was created during a racist era. He was drawn as a caricature of a minstrel which itself represented a demeaning image of blacks. There is racial stereotyping of black people in Florence Upton's books, including The Adventures of Two Dutch Dolls such as the black minstrel playing a banjo on page 45. It appears that the Golliwog was another expression of Upton's racial insensitivity. Certainly later Golliwogs often reflected negative beliefs about blacks thieves, miscreants, incompetents. There is little doubt that the words associated with Golliwog Golly, Golli, Wog, and Golliwog, itself -- are often used as racial slurs. Finally, the resurgence of interest in the Golliwog is not found primarily among children, but instead is found among adults, some nostalgic, others with financial interests.