Yes, they were being passive aggressive, but your behaviour was also not great.
You should have reasonably factored in potential delays:
Your son gets carsick, so needs a halfway break = add 20 minutes.
You want a sandwich = add 20 minutes
The dog needs a walk/wee = add 20 minutes
Saturday traffic in August = add 20 minutes
The fact that you were seemingly oblivious to all these factors, and stroll in 40 minutes with an "oh well!" attitude... They were, not unreasonably, a bit annoyed.
They held off the annoyance, because, "well, they're here now, might as well crack on with the day", and then it erupted.
But you know what? People behave oddly all the time. Yesterday teatime I went out for drinks with some work friends. Organised by the ringleader, we'll call Jenny.
We'd been there about an hour, people ordering drinks and food to the table as and when they wanted it, all chatting and laughing.
Then Jenny just announced "I'm going to take off now." and within 30 seconds, she was out the door.
The remaining 4 of us just looked around, a couple of "that was a bit abrupt!" comments, then we shrugged and continued enjoying our evening.
People have all sorts of complicated things going on, and do things that can sometimes seem strange. But all you can do is shrug and carry on. Not everything is a personal slight.