IME when people smile at my son (he is half Chinese and half white), it is because he is cute! I honestly don't see anything sinister behind the smiles. Chinese people make a huge fuss of him and ask lots of questions about his age and weight and teeth, and English people are more reserved with children I find, but they always ask how old he is etc and comment on how long he is!
DH and I have spoken about racism before and he said he has never had any trouble at school or work or in the street. Maybe he is lucky, or maybe he is just an easy going person who likes a laugh. He is friends with people because he gets on with them and that includes white English, British born Pakistani, British born Indian, Indian, Somalian, mixed race white English and Philipino, mixed race white English and Japanese, mixed race English and Thai. The only person who has suffered racism is our British born Pakistani friend and I think that is because he lives in another town nearby.
This is not to say I haven't seen or heard racism because I have, mainly at work from 20 odd year old boys who are very immature and complain mainly about Muslims and also spout DM style crap about Polish people taking all the jobs
I think a lot of this is to do with the area they are from, there is a town nearby which has a large Muslim population and a growing Polish population, coupled with low unemployment and there have been tensions historically and I think people just copy what their families and friends say without really thinking about it.
I think it's best to acknowledge that yes, some people are racist or just plain stupid, but please don't think everyone is racist because they aren't. I think most people are nice, it's a shame it's the horrible ones that stick out and that you remember.