You did brilliantly - polite and calm, perfect. You have no reason to engage with him now apart from on your terms.
He doesn't like the fact he no longer has an upper hand in this situation and it's thrown him. Tough. Now he will get a turn at having to second-guess situations and question himself. See how he likes it.
Keep hold of that cool exterior when he brings the children back and in all future encounters. I know you want to scream and cry and demand answers. That you want to know how he can behave like this towards you and the children. But you won't get a reasonable conversation nor the answers you want so don't open that can of worms.
You want him to break down, say sorry, admit he's a fool, beg forgiveness. It ain't going to happen. It might happen in the future but certainly not yet. Honeymoon period with the new woman and all that. Try not to fantasise about him doing that because it will drive you mad.
Staying calm with him also means he can't use your anger as another reason to justify why he left. Of course it's entirely reasonable for a person to feel angry when their spouse walks out but right now you need to play your cards close to your chest and not let him see that. It'll really unsettle him.
I hate game playing in life. But in this one situation I think it's exactly the thing you need to do to stay sane and stop him using your actions against you. He's had this all his own way and now you are taking back some power and control to protect yourself and your children.
Be proud of yourself. You are doing amazingly.