This truly isn't a vegan thing, it is that culturally, the British despise those who they perceive as puritans. Many people would feel deprived without meat, eggs and dairy, and so they perceive those who voluntarily deprive themselves day in day out as ascetic, and therefore, suspect.
See also, to varying degrees, teetotallers, those with religious beliefs of the "god-bothering" rather than social ritual variety, frugal people, socialists, people who exercise, people who work too hard, people who are too houseproud...