I agree with Lweji, a lot of the adverse reaction is to do with the way these things are explained/discussed.
Only a ninny would have a problem with someone who said 'I'm sorry, it looks delicious, but for medical reasons I'm not able to eat X'.
The problems come in the other cases mentioned on this thread: people who claim to be vegan/vegetarian/gluten-free/sugar-free etc etc but in fact only follow that way of eating when it suits them, and the instant they see something they like the look of it doesn't suit them any more. Hence, hosts who've put themselves out and incurred extra expense to cater for what they believed to be a diet followed for ethical or medical reasons discover that actually it's just a fad.
Or the people who bang on endlessly about why it's a bad idea to eat XYZ while others are tucking into said foods and trying to enjoy them. Or those who explain in far too much medical detail why they can't eat XYZ, again often while others are trying to eat.
Of course, it's clear that there are a lot of ninnies about. Ignorant people who struggle to believe that allergies and intolerances are real and ignore what their guests/customers say about them, or even actively try to expose their guest to the allergen/non-tolerated food in the hope of catching them out or in the criminally reckless belief that it will 'help' them because they'll never get over it if they don't just keep eating that food until the body gets used to it.
Or the slightly less irresponsible but still deeply unpleasant people who deliberately feed a vegan/vegetarian with food they know isn't vegan/vegetarian.