When ds1 was under 4, we visited some childless friends for sunday lunch. Ds1 got down from the table after he had eaten his main course, and the host told him that he had to stay at the table if he wanted any pudding. Ds1 got down because he was bored - the adults were chatting over the end of the main course, and he wanted to amuse himself with some toys rather than just sitting at the table waiting for pudding.
Because he got down and didn't go straight back to the table when the host told him, he didn't get his pudding. Inside I was a bit 
, and felt that our childless host didn't understand small children, but outwardly I backed him up as I felt 'his house, his rules' applied. I wouldn't have texted him later to complain either - ds1 survived the terrible pudding deprivation, and I felt the friendship was more important than making a fuss about something so trivial.
When we have guests here, I tend to be fairly relaxed about 'the rules' and generally wouldn't want a guest to feel that I was criticising their child's manners or their parenting. I have made the odd exception to this - for example, when a friend came to stay with her two sons, one of whom wanted to come to the dinner table for sunday lunch stark naked! At that point, I did say, 'No clothes, no lunch' - and felt that was entirely reasonable too! My friend didn't object either, so I can only assume she felt it was reasonable of me to insist on no naked diners.
Later on, the same child refused pudding, so I dished up pudding to those who wanted it, sharing out the dish with no left overs. He then decided he did fancy dessert after all, and my friend shared hers with him - and that was fine by me (though I wouldn't have shared my pudding with any of the dses if they'd done the same thing - I would have offered some fruit or yoghurt).