I get where you are coming from OP - if you left her baby sitting and with a timetable of the way things worked as you like them, and she didn't say that she was planning on doing anything different, why would you expect her to go gallivanting off all over the place? And if she did go out then maybe you'd think she would pop down to the park or to the supermarket to pick up some more milk - something local rather than something hours away...
I've had the same sort of thing happen to me - a babysitter when the dc were 5&2 looking after them on a sunday afternoon when we were out - instead of staying at home and doing all the things to do at home that I'd left for them (including ds1 wanting to watch the world cup final) decided to take them out the moment we had left, meeting up with her dh and dd (a teenager) whilst out and playing pretend happy big families (think she always wanted more dc). She didn't bother to mention to me that she had planned to do this (I would have said NO - I wanted them to have a nice afternoon playing at home for a number of reasons, not least I wanted them to have an afternoon at home and i didn't want her driving them around without any car seats doing her jobs whilst I was paying her to look after my dc).
Who expects somebody who is babysitting to go out with the dc? You don't. I know lots of babysitting happens in the evening so the dc are about to go to bed etc which makes it different - but even so - it just never occurred to me that if I was paying someone to babysit that they would take the kids out and about. If I was using a childminder then I would have different expectations - but a babysitter - I would expect them to stay at home!
So would definitely be very upset even if it was my mum and she had taken them off for the afternoon.
Having said that, I'm surprised you didn't say when you thought you would be back - I bet she just assumed that you would make the most of being away and be back as late as possible because that's what she would do (or thinks what she would have done). Whereas it's obvious to you that you would be back much sooner for your dh to be able to spend some time with the dc... Unfortuanately I think you need to chalk it up to being an example of when everybody has assumed different things because they were so obvious to them that they didn't need spelling out whereas actually they weren't obvious at all and should have been spelt out explicitly before you left.
Sorry that your dh is being so grumpy about it (or he was when I started typing this, got interrupted so the thread could have moved on a bit) - he should realise that it wasn't done on purpose, he could have said to your mum about seeing her tomorrow morning when he left yesterday if he wanted to come back asap so he had a chance to have stopped the confusion too, it wasn't just you and your mum responsible for the mix up. He should have grumped, said that next time you should both make a note to make sure this sort of mix up never happens again and then have a nice afternoon together rather than sulking the rest of it away!