You son is at the age where he'll be places without you (like school) and have his own relationships with other people, and people will tell him off sometimes. Kids soon learn the differences between adults - what one parent / grandparent will allow and another won't, which teachers can control a class and which will let them mess about etc. Best to just accept that and let him get used to it and accept that there's different rules in different environments, without fussing or crying.
Maybe it's time to get stricter yourself, though. There's too many mothers who are too soft and raise boys to believe that only men are to be respected and obeyed, while women may be safely ignored. You don't want to raise a sexist man. Instead of teaching him that he has to do as he's told the first time when he's at granddad's house, if granddad says it, why not teach him that he has to do it when YOU say it?
Although having said that, if your father was abusive, maybe it's time to think seriously about whether you're going to benefit from a continued relationship with him - and whether your son will?
If you do want to keep seeing your father, maybe it's time to work on standing up to him a bit better? I mean, what's your son learning there? Man shouts, woman backs down / runs away? Men are in charge? He who shouts loudest gets his way?
You sound like you're scared of your father - that's what needs fixing. You also sound like you're scared of acting authoritatively with your son (maybe because of how your father was, you've gone too far the other way?). The one time your father corrected your son's behaviour at the table (which wasn't acceptable, at five) isn't the real issue.