I'll tell you a story which may or may not be relevant.
A friend once said the same thing to me. She asked how to cope with all the hostility she was getting about her five year old being home educated. People looked at her funny, she said. When they asked why her son wasn't at school and she replied he was home ed, they changed the subject or the conversation stopped abruptly.
I was astonished, because my experience had been the opposite! Almost no one I spoke to was really negative. A few were positive, some showed little interest or made a polite neutral remark, some had tentative reservations or concerns which seemed to melt away when I explained a bit about how it actually worked.
What made this so remarkable was that my friend's child and mine were the same age and we lived in the same area. We rode the same buses, went in the same shops, visited the same libraries. Our children were both fairly average and behaved like typical kids. Why were people so unpleasant to her but fine with me?
We talked about it some more, about how these conversations developed. It turned out that my friend had an expectation that people would be critical. When she was out with her son, she would get a knot in her stomach at the prospect of someone giving her a hard time. She hoped the subject wouldn't arise. If it did, she quickly answered, "he's home educated" in a clipped voice with a hard look on her face, a don't-even-go-there look. And I think that is WHY other people "didn't even go there". Instead they silently wondered what was wrong, why she seemed angry, why she didn't want to talk about it, and maybe they gave her funny looks as a result.
Then there's me. I was, and still am, something of a home ed evangelist. I think it's the best thing since sliced bread, and that many people would love it if they had a chance to try it. Whenever someone asked, "No school today?" I'd respond, "no, she's home educated!" with the same expression I'd use to announce it's my child's birthday today. If anyone asked a question about HE, I would cheerfully chat about it. It was clear that I expected people to be happy for us, so usually they were, or politely pretended to be. Maybe that was because I seemed to know what I was doing, or maybe it was because they wanted me to stop rabbiting on about it. People will often take their cue from you.
So if you think a similar dynamic might be affecting your interactions with people, how do you turn it around? I don't think you have to transform yourself into a chatty pest like me. But if you're happy and quietly confident of what you're doing, people will often follow your lead in the conversation. If you don't feel that confidence yet, it may help to spend more time with other home educators, either in real life or on forums or by reading blogs or books. Their experiences and attitudes and knowledge will rub off on you and buoy you up. Then you won't worry so much what a clueless stranger thinks. Ironically, some of them will be more positive as a result of you being forthright and unafraid of their opinions.
Does that seem like it could be what's going on for you?