I have occasionally had this. In my case, I have noticed that it often comes from people who are really quite weirdly attached to food in an emotional way, who couldn't contemplate in a million years going without meat because 'they enjoy it'.
Some people really believe that logic, or think that animals don't have feelings or that they can't feel fear, and fair enough. However, in a lot of cases, these comments often seem to come from a deep sense of discomfort with the hedonic logic that this implies. They will often be decent people with quite a strict sense of morality - though, as with most of us (myself included), this is more often activated when it comes to passing judgement on other people's behaviour than on their own! Also, they are not stupid so they see the problems with the defence that anything goes just because you happen to enjoy it!
Generally, people don't like the logic of applying morality to food in particular or consumption in general, not because they disagree with it but because their whole moral code would lead them to agree - but that would be awkward and demand that they take responsibility/make choices. Vegetarianism kind of challenges that, even if you never mention it except when you absolutely have to (e.g. work meals). I get how that feels - I have a friend who told me off for buying something with palm oil in it, and I got really defensive about it at first (inside) - because I totally knew she was right and that I should walk to another shop to buy an alternative that didn't contain it, and to be honest I felt guilty and ashamed that I hadn't considered it. She was right, but I didn't want to acknowledge it.
The positive thing is that sometimes this openness means that if you actually react calmly and in a friendly way, without being horrible, judgy or sanctimonious, you can actually get an audience for a meat-free diet and even convince some people to cut down or go meat-free! Not ramming it down people's throats so that you earn their trust is important, though.