Should we have said anything??

[Imissmetime]

I am so conflicted about this and just want some thoughts on whether we did the right thing.
Currently on holiday and was out with friends and the kids at the seafront yesterday. We stopped for chips in a crowded area and another family were stopped in front of us waiting for an attraction. Man, woman and 4 small kids, all I would guess under the age of 8. Anyway, one of the children about 5 was playing up and the man ( who looked. Slightly manic), started screaming and shouting at her for misbehaving, he Grabbed her arm roughly and twisting it behind her back leaving visible marks even from the distance we were at, he then said he was taking her away and threw her over his shoulder which was the only time she cried and liked petrified ( they came back in a less than a min) but he continued to grab her arm harshly and a scream at her. The mum said nothing even though the man looked out of control. Weirdly, the child never cried even with the awful behaviour to her ( maybe too familiar with it). Like I say it was a crowded area and I saw a lot of conflicted people. Should I have intervened?? I didn't out of fear for making it worse, but feel that if I had the child would have known that at least people thought it was an unacceptable way of behaving!! The situation was diffused with the man walking away for a bit and the child sitting as still as a statue. We then left.
YANBu- you were right, it would have made things worse if you intervened
YABU- you should have said something (if you think this please tell me how I could have started that conversation for future reference)

So hard to know....l want to say yes l would intervene but having been on mumsnet a while now, l think some people say you would only have made the situation worse for the child.
In all honesty l think l only would have said something if l had my husband with me as he would be there is the guy turned nasty on me.
Poor little child it is awful to think someone could do that to their baby

Thanks for your feedback, and yes it really is so awful how someone can be like that to a child, our kids were so distraught at watching the injustice,and on their behalf I felt I should say something but was so worried that the child could be punished afterwards for making a scene.
Genuinely interested in hearing from someone who has any experience in the area about whether I should Intervene in future (I appreciate its never black and white but any guidance appreciated) or does that make it worse???