Goodness me, sounds miserable. Bored, hungry, overtired kids, and driving a long way home late in the evening - it's not fair to mess you about like that.
Next time, if you're prepared to visit again after this, maybe try & take control a bit so you're not kept hanging around if they're faffing about. For example this time if I understood it correctly you waited in their house for them to come back so you could all go to the restaurant?
Next time, arrange to meet them at the restaurant at say 1230, if they don't show, you can still have your lunch at a civilized time, then wait a bit for them if you choose, or just leave.
Alternatively, if you've got the nerve, be very clear with your husband that regardless of what happens, you are leaving by X time, and actually do it.