If you label yourself and make a drama about what you eat/don't eat and virtue signal your way through every mealtime - you'll piss people off. If you just eat - as I do - no-one even notices.
And here we have the problem. Simply being vegan (which you have to mention when people ask why you don't want milk or whatever), is seen as 'virtue signalling'. People always assume they're being judged when someone has a different or more extreme ethical standpoint than them.
I'm vegetarian, and if I've ever mentioned it (last time was to explain, in response to questions, why I was turning down marshmallows), I've had everything from "But bacon!" up to people trying to force meat in my face. Last time, the marshmallow time, the person involved looked disgusted and said: "At least you're not vegan, though!"
It's a bit like if you don't drink and people are always asking why, or trying to get you to drink, or talking about how good it is. It's practically impossible to not eat meat without regularly being asked "Oh, are you vegetarian?". Which is weird as you'd think it wouldn't be that unusual to occasionally have a meal without meat!
It's exhausting, frankly. I have friends who are vegan but if they're with people they don't know they pretend to be allergic or not hungry, just so they don't have to have these conversations.