I know people who have bought the book "Mein Kampf", by Adolf Hitler, and have it in their bookshelf. Would I assume that they are nazi apologists just by that fact? No, I don't assume anything, as there are many reasons they may want to read that book without sharing the ideas in it; in fact, there are many people who choose to read books written by people of different ideologies, or even by fascists, in order to gather a broader understanding of the minds of those who perform atrocious acts. Or he/she might have that book because it belonged to a nazi soldier who died in WWII and his grandfather picked it up and kept it. or he may just be curious as to what Hitler had to say. I would not expect that person to have to hide the book or dissolve into explanations of why he/she is not a nazi in front of every stranger. If I was curious to know, I would ask politely. I definitely would not be up in arms telling him that, if he has that book, he has to be prepared for people to assume he is a nazi. I tend to think that people in general have to be prepared not to assume, and not to be offended at things that weren't intended to be offensive. I also tend to expect good in people, unless they show otherwise.
If a person I knew was known to be dismissive or aggressive towards black people, if he or she made offensive remarks, and on top of that, displayed a golliwog on his/her window for everyone to see, then I may suspect that person might be racist (but in this case, I wouldn't have needed the golliwog to reach that conclusion). If a random person who I have no reason to believe is racist, decides to buy a golliwog, then I can't imagine myself thinking: "Gosh, he/she must be a racist. I will not speak to him/her ever again!".
Once again, this is the way I am, and do not expect anyone else to be like me. However, I am quite convinced that I am a much happier person, less stressed, and much more confident in my own self than most of the people posting in this thread showing extreme offence towards someone owning a golliwog. Oh, and somehow I am also convinced I am much less racist, perhaps by my experience that people who don't have prejudices do not feel the constant need to shout and rant against them. People who always see the straw in everyone else's eye are usually blissfully unaware of the rafter in their own.