Actual intollerance stuff like sandwiches on gluten free bread, dairy free etc should be kept separate from the "main" buffet to avoid cross contamination, or ideally a plate made up, and wrapped/named.
Don't agree about vegetarian/vegan options though. That is really more a preference than a "need" as such, you choose not to eat meat/animal procucts rather than physically can't (in most cases) so it's really unfair to gatekeep anything that happens to be veggie/vegan from those who do eat meat/dairy.
I'm not vegetarian, but i am incredibly meat and fish fussy, with a majority meat free diet but lots of dairy. I won't eat tuna, and i don't eat any kind of cold cut meat, don't like egg mayo, so the majority of buffet type sandwiches are out. The only sandwich option i would eat is cheese, or the other veggie/vegan options. I don't eat sausage rolls, pork pie, cocktail sausages etc that are the other "typical" buffet foods, so really, the veggie/vegan stuff, is the only stuff i would eat.
Maybe the non veggie/vegan colleagues of yours are also fussy eaters and just don't like some of the sandwich fillings, so take the cheese/veggie options as thats something they do like? They don't write they have dietary requirements as technically they don't, but that doesn't mean they will like the meat options offered.
Honestly, do you think it is any more fair for someone who eats meat to be forced to eat the meat options they don't like, or not eat at all because the veggie/vegan food is reserved for those who ticked a veggie/vegan dietary requirement, and nobody else?
Ideally the caterers would just let everyone specify what they want, and everyone would take what they specifically asked for, or you could ask them to provide all meat free/lower portion meat bigger portion meat free. Either way, it's not the "meat eaters" fault if theres not enough none meat food provided to go around everyone who wants it.