You moved out; you don’t live here anymore. I am not ‘stopping you from seeing your children.’ You can choose where you take them, whether it’s to your new home, to the park, to your mum’s house, to soft play, or to a coffee shop. We are exes now and I have moved on, and that means that you are no longer allowed access to my personal living space
This. Exactly. He does not get to 'pop over' when it suits him. You are NOT being controlling, you are providing the greatest amount of stability that you can for your kids. Those two things are worlds apart. Providing stability is important in any busy family, but even more important following the chaos that he has caused, and the upset that has (and will continue) in their lives while they adjust to the new situation.
You cannot provide that stability if you can't prepare your dc for what is happening e.g. "yes I understand you miss daddy, but he is coming after work on Tuesday to take you to the library. Then he'll drop you back here and I'll read you a story before bed."
Every time your ex calls you controlling, replace it in your head with the phrase 'providing loving stability and certainty for the sake of my dc'.
Perhaps you could tot up how many actual hours your ex has spent actively parenting your dc over the last couple of weeks, and say that until he can implement the new arrangements properly, he gets no more time than that.
My heart goes out to you during this difficult week. Like other posters, I suspect that once he has actually moved out, your dc might be calmer - there's no more wondering whether daddy's going to be out all afternoon, or if he'd coming back in time for bedtime.