I think yabu. He was probably hurt and a bit embarrassed because you didn't explain properly (from the sounds of it) and didn't want to risk being told off again. The three hour drive and the hot day aren't really relevant as far as I can see.
I might have done what you did (regrettably) but then smoothed things over once I'd gathered myself together and apologised for coming off rude, but explaining how she needed some time to come round otherwise it'd be hell for us all.
If that's how you feel, then you should have had it out with him, rather than leaving after 3 hrs! (I presume you intended on staying for longer, considering the journey you undertook to get there?!) Just as childish and churlish as him IMHO
You are being completely unreasonable. He came over to see his granddaughter and you told him to back off. I would've avoided going near her again after that without a better explanation or apology for saying it in a rushed manner because I'd be worried about being told off again! And I'd be concerned it would be embarrassing if I asked if it was ok to talk to or play with her later on in case you said no again.
I think a quick "she's just woken up, she'll be grumpy for a bit" explanation doesn't take long and would have sorted the problem before it became one.
Did you tell him when he was allowed to talk to her? Or did you just tell him to leave her alone with no explanation? because the poor guy was probably too nervous to approach her after you snapped at him.
You know, I'm probably allowing my own FIL issues to cloud my judgement here...! In a way YANBU, but, could maybe had a little more tact...In the same situation with my larger than life bully of a FIL I would have said nothing, allowed my child to be rudely awakened then been on here asking "AIBU for being such a doormat?!"