Stay strong
He's playing the grand gestures to get the kids on side, but it's just that, gestures.
At the moment, he's the fun one and you are the naggy one, the one that gets them to bed, make them eat decent food at reasonable times, gets them dressed and to school. It won't take long, especially for your eldest, to realise you are the good one, the one that keeps them safe, that will always be there for them, the one that truly loves them for them 
Do you think he was such an arse with the baby so that, next time, you tell him not to bother, just take the older two?
Arrange next week, say it didn't work having them back so late when they have school the next day, especially when it was later than agreed. Factual, not emotional. Even better if you do this tomorrow, ask the teachers how they've been in school and see if the grand day out has had an impact on their behaviour / attention - more facts for you if necessary. Suggest he either he does Saturday 1pm to 6pm, so they have time to recover the next day from a big day out, or he does from 10am to 3pm, so they can wind down afterwards.
You can do this. You ARE doing this!