I do find it annoying, yes.
There are foods I really don't like, e.g. olives and raw tomatoes. I would always pick these things out of a salad, or sandwich for example, rather than eat them. It is definitely not a choice I am making to be fussy, because raw tomatoes are in loads of things and it is a pain not to like them. I have tried to force them down in the hope of growing a taste for them, and it hasn't worked. I still don't like them. Same with olives.
However, with both these foods, I would and could still eat them to be polite, especially if they were just one aspect of a dish. Like when someone has made a bruschetta with tomatoes and other flavourings and things, I will just eat it, or at least some of it. I like bread and the herbs and flavourings, and they mask the tomatoes to some extent. Same if someone has served up a pasta dish with chopped olives in it, or a tapenade. Much as I dislike olives, I can still manage to enjoy the other elements in the dish and just grin and bear the olivey parts of it.
Therefore I do find it annoying when others are really fussy to the point of going all quiet when a meal is served that has a food they don't like in it, and then not eating it at all because of said element. As a cook/host, I also find it annoying when a conspicuous pile of said ingredient, e.g. olive or tomato or whatever is left on the side of the plate, if it is a dish I have spent time and effort preparing. I wouldn't say anything about it, because I do recognise that people have foods they would rather not eat. But I do internally find it annoying.