Any dinner party-type threads on MN are baffling/hilarious/revealing in equal measure.
There was one a couple of years ago, where someone was annoyed because guests invited for 7 turned up at 7.15 pm. No cries of le quart d'heure de politesse would change their minds (or half the thread tbh) that their guests had been disgustingly rude.
They had said dinner at 7pm, so it was banged on the table at that time. At a dinner party. The late guests (who probably thought they were being perfectly polite) missed the beginning of the meal. They thought dinner party rules applied - being greeted with a drink, maybe canapés, moving to the table at 7.45 ish, that kind of thing.
But this isn't a dinner party. I can see both sides here. I would never turn up terribly late without a phone call, unless everyone was aware of the need to be flexible because of work. But the OP wasn't terribly late, and I would find it slightly odd in those particular circumstances to begin the meal until all the guests were there.
I agree with a pp that it's about the family gathering and sharing the time together, not about that meal being on the table at 6 sharp.
Personally, my pet hate is anyone turning up for dinner early. That would REALLY annoy me 
If someone invites you, and says "meal will be around 5 - 6" then you should turn up for 5, in case it is ready then
I would never interpret that message in that way. If I told someone that, I would assume they might turn up any time between 5 and 6 ish, and we would eat after chatting for a short time, maybe having a quick drink or something.
I'd actually be surprised if they turned up at 5 on the dot, 'in case the meal was ready then,' expecting me to slap the meal down on the table at precisely 5 pm 