He can't say it doesn't suit without giving a reason. It might be money, but he's going to have to swallow his pride if this is the case, and be honest for the sake of the kids. Moneysavingexpert.com has lots of good advice about how to get train tickets very, very cheaply if you're booking far enough in advance.
TBH, I think your mediator should have seen this coming. You're doing a lot of travelling with two very small children, I don't think this should be happening every visit. Whoever has the children the majority of the time is already doing most of the hands-on stuff for the children.
You've trialled the idea you both came up with, it isn't working for a number of reasons. Time for a new solution. You could put this all in a letter to the dad (ie the effect on the children of the journey, the expectations, the after-effects of it all, and his lateness giving dates and times so it doesn't sound vindictive.) Suggest your own solutions within that letter. I don't think he'll agree to change it while you keep going along with the current arrangements.
It's the kids who should be at the centre of this, and they clearly aren't responding well. You've tried it all more than once, that shows your willingness to stick to the agreement.
Good luck! You don't need to do this anymore.