YANBU at all.
Our tesco delivery a couple of weeks ago came early, but the driver rang me first and asked if it was okay to bring it early as he was passing. This was fine by me so I said it was okay.
The next time, they were on time and the driver was whinging about the fact that he had driven past the end of my road half an hour before my allotted time but he wasn't allowed to deliver then as it was against their rules. I said that he should have rang me like the last guy and he said that they are ABSOLUTELY not allowed to do this.
He said that they absolutely have to stick to the times stated otherwise they get in big trouble, but the people that organise their routes don't really have a clue about where things are and a realistic time of getting from here to there.
If I were you I would be very tempted to complain about him. It sounds like he was fed up and pissed off and trying it on and you were the victim. He is not allowed to do this.
grr