"Careful" definitely annoyed one mum last summer.
I said that as I caught her 4ish year old by one arm and held him firmly whilst he wriggled and kicked. He started to scream (he had a better repertoire of swear words than I do) and kick harder, I could only grip ever harder and hope his mum/someone/anyone arrived quickly.
She did, shouting and swearing at me, waving her arms around, asking for someone to "Get that fucking woman off my kid".
Oh, forgot to mention, I was stewarding our Open Day, wearing a Security tabard, and was holding her kid to prevent him from walking through the equine yard entrance as about 20 horses were trotting through into the arena for the main event. He was waving his arms and shouting "Horseys, horseys" which was, understandably, spooking a few of them.
Mum got even louder as they were escorted off the premises. Her baby boy loves horses, you see! He just wanted to see them. I'm not sure how much closer he could have got without getting squished. He'd have seen them from very close quarters if I hadn't grabbed him.
She wrote to the local paper to complain. But they ran the story from our perspective, though they did include a beautiful DM sad face of her and baby boy

I know it isn't the kind of thing you meant, OP. But sometimes inane words are all you have!