YANBU OP.
My friend and I were in town a few years back when our dc were about 3. We went for lunch in the only pub that had "family area with kids welcome" in the whole town. All the other pubs didn't allow children.
We sat out the back in the "family area". We were waiting quite a while and the 2 dc got a bit whiney as they do. My friends dd cried and we calmed her down.
A man sitting opposite us (old couple in the family area, no kids) shouted over "your child is spoiling my lunch, you NEED to make her be quiet". My friend is very shy and just blushed so I said back "sorry this one doesn't have an off switch". He then started moaning saying pubs (family pub at 1pm btw) were not for children and shouting "get that child out, off to the playground, out out out!" he was going nuts! So the manager came over and he said "I want that child removed, now!" "get that child away from this pub, they are spoiling my lunch" to which she told him he was sat in the family area and children do cry sometimes.
By now I was sick of his shouting which was making the dc and my friend really upset. So I went over and told him he was a miserable old bastard and to fuck right off. Then sat back down. He shut up after that. As they got up to leave his wife came over and apologised for her husbands behaviour and said she knew how is was to have young children. I told her that she was just as bad as him for sitting there not saying anything and should be ashamed of herself for listening to her husband intimidate two young mothers with their small children.