YANBU. I think it's practically one of the worst things about allergies, other people poo-pooing it and not taking it seriously. My son is severely allergic to cows milk protein, also allergic to egg, lentils, peas, nuts...... When he was at nursery school they just seemed to be so dismissive because the problem was milk - they kept telling us they were very experienced with allergies as they'd had peanut allergy kids there before. They INSISTED that he had school dinners, and insisited that they could cope.
During his time there they gave him: Chocolate shortbread, quiche, and finally the last straw, A GLASS OF MILK. And this was an Ofstead 'Outstanding' nursery.
That is the worst that has happened to him at the hands of other people's ignorance, but I do feel worn down some days by the little things like, friends buying him a chocolate easter egg (he can have chocolate though can't he?) and both sets of GP's in the beginning not taking it seriously at all, and thinking we were being precious. If only we'd let him eat some mud, or try their dinner he'd be fine....
Eating out is always interesting too. Last time after explaining to waiter that he could not have anything dairy, and listed all the elements, milk, butter, cheese, egg etc etc he then went on to suggest that we give him 3 dishes which contained cheese. :S
Most of the time we do take it in our stride but, yes,occasionally it makes you fed up! :)