I get it and whilst I understand that you are hurt and it is worth raising that you would have been open to adjusting the timetable had you know that he was leaving a day earlier but I think you would be unreasonable to make a big thing about it.
I didn't like my paternal grandparents as a child. I understood them better as an adult. My nan was a hard woman to love and I didn't really know my grandfather. They lived separate lives within the same house and he had no interest in any of the grandchildren even to the point of not knowing our names. My nan smoked like a chimney, on more than on occasions I witness her smoke 20 cigarettes before I had finished my breakfast. My grandad went to the pub every day for 3 pints, more on a weekend. He would be probably classed as having a drink problem in modern life. They were fine, they weren't loving and nurturing like my parents or my other grandparents but actually some of my fondest memories of them are the ad-hoc days/nights they had us without parents around. They seemed to step up on those days.
On the driving front, where there's a will there's a way. I have a clumsy family and we have managed to get to A&E multiple times without a driver in the house - taxi, friendly neighbour, ambulance, walked, other relative.
I would wish him a happy holiday but also let him know that you would prefer to have been told about the change by him and would have been happy to pick them up earlier - maybe suggest that after having had the day with grandparents you could have got the for the night?
These situations are messy and non-binary. It's less reasonable/ unreasonable but lot more nuanced from that. It wasn't unreasonable for him to make the arrangements but also it's not unreasonable for you to have wanted them in preference to the grandparents. Also, see what the children have to say afterwards