The thing is, other people have feelings too. You aren't the only person who has emotions worthy of consideration.
And it's more complex than 'someone doesn't want to eat your food'. If someone doesn't want to eat my food, fine, decline the invite. I have Orthodox Jewish friends who won't eat food I've cooked, but I know what they will eat (things they've brought, fruit, etc) - and I'm fine with that, they can visit for the day or even stay overnight and we all know where we stand.
But if I invited someone to eat at my house, and they they came and then, without warning, they didn't eat anything (unless it was a big party with lots of people milling about so it wasn't completely obvious), I would feel... confused, upset, irritated, I'd ask if there was anything you'd like, I'd feel rebuffed, I'd look at the food I cooked for you and wonder what was wrong with it (I'm a good cook). I'd also wonder why the fuck you hadn't told me that you had a long list of food issues when you accepted the invite.
Not everyone is going to understand that ARFID is a thing (I didn't know before I read this thread, and unless I encounter it again within the next six months I'll probably forget it).
So if someone doesn't know about or understand ARFID, and has gone to the effort and expense of planning the menu, working out the quantities for X people, buying the food, preparing it, setting the table, cleaning the house, and all the other hassle that goes with having people to dinner, and there is someone who has come to dinner, and who eats nothing, then they're justified in wondering if mind games are being played, or worrying that their food is gross. And being anxious that they have a guest who is clearly unhappy, and feeling guilty about it.
They're going to be baffled and pissed off by the end of the evening.