I agree to bide your time, for someone who is off living his new shiny life that he wanted, he seems to be a very angry man. I'm sure he's already realising the grass isn't greener.
Of course your argument at mediation won't be he's not sticking to the times agreed as such, it will be that he is distressing the children bringing them back at a time unagreed and allowing them to think you have forgotten etc. This is abusive not just to you but particularly to the children, the mediators in my experience were less concerned about impact on me, and only concerned about impact on the children. You therefore are unable to go about your life in case he does bring them back - it's simple control. He's lost control of you now and doesn't like it.
You're doing amazingly well, wishing you a speedy recovery.