Something similar happened to us when we were visiting the PIL. I was coming down with a D&V bug, so wasn't as on top of my toddler as I should have been (too busy running to bathroom to puke!) and he was being a little naughty (he was 2 at the time, so nothing major). Normally, we would head home once he was getting to the tired and bored stage, so would avoid this behaviour.
While me and DH were getting ready to leave (DH only just having arrived to drive me home), FIL lost it with my DS, and lashed out at him. It was only because my DS turned away at the last second that he wasn't walloped! FIL's face was red with rage, so he had clearly lost it.
We were too stunned to say anything, and just gathered our things and left. My DH was a bit like yours, and has been conditioned not to cross his father, but this was too much even for him to ignore.
The next time we went there, we dealt with it as low key as we could. My DH took my FIL aside as we were leaving, and had a word with him, to the effect that he was not to hit our DS, and that was not how were were dealing with discipline. He seemed to take it well, and we thought that was the end of it.
We came to find out weeks later that FIL was refusing to come to ours and see us (no big loss, but it meant MIL, who can't drive, couldn't see our DCs either), and was in a big strop. All because my DH had the temerity to say anything to him. My MIL was sticking up for him, and basically denying that he almost hit our DS. Apparently he was just 'getting out of his chair'. Yeah, right. It all culminated in a phonecall that I had with my MIL (as my DH was not really dealing with his parents at all well), where I told he what I really thought, without mincing words. Not pleasant, but far too much pussy footing around my FIL for my liking.
I'm not sure if this helps you at all, but you're not the only one in this position, and to my mind, it doesn't matter that you were in their house, your FIL had no right to get that physical with your children. He should have walked away, or told you to do something if he didn't like their behaviour. He lost control, pure and simple.