Oh my goodness. I can so understand how you feel. And I really admire you for coming on here to try to sort your feelings, rather than just allowing them to overtake you / allowing the children to see them.
I really, really think you need to sit down and work out a plan for how you are going to talk to your ex less. He should not be lingering in your home, asking you things. He should not be telling you of the plans he has with the children + OW. That is HIS life. It has nothing to do with you.
It sounds like he is looking for your approval, and/or he gets a sick ego boost from sticking the knife in by telling you of his plans. It's unfortunate but many men blather on like this and seem to have zero insight into how it makes others feel.
Some tips that I've had to use with my ex, who I had to cut out of my life as much as possible:
- Write a 6-month long spreadsheet of when you have them vs. when he has them. Include exact times for pickup and dropoff. Have him initial the rows where his times, including pickup arrangements, are detailed. You can use Google Sheets to do this.
- Arrange handover of children at school, or a neutral location like a playground, or, at worst, he needs to collect them from the doorstep. He may not come into your house. Notify him of this nicely, but briefly and in a businesslike way, via email: "Hi Ex, I'm sure you will agree that the interests of the children's mental health, it is important that contact arrangements are consistent and predictable. In this vein I want to reset expectations for when you collect the children. In future please collect them from (detailed description of where - including "from the front door of my home" if needed) at the agreed time in the contact spreadsheet (include a link). If you are not able to collect them at the agreed time, we will wait for (x) minutes, after which time your visit with the children will be cancelled and notated as missed contact on the contact spreadsheet." -- something like that.
And reply or drama that comes back from him, ignore it. Until he asks a question that you can answer with a fact, like what time, where, etc., ignore ignore ignore. He will quickly get the gist of things.
If he does attempt to talk about OW plans, respond with silence. No questions. No anything. If he keeps going on: "OK, must be going. (kiss children, start to shut door) Have a nice time!" (shut door).
Do not allow space for him to talk at you about his new relationship. If he asks you where to take the children, respond with a puzzled look, and after a pause: "I don't have any suggestions. Perhaps you could Google it? Have a nice time!" (shut door).
This will be a slog for you because you are still emotionally very raw and, of course, trying to learn how to completely disentangle yourself from this person you thought you knew. I completely identify with that. It is very hard. Be gentle on yourself and recognize that you're going to need a lot of practice and time to get used to this whole new world you're in.
It does hurt. It's OK to hurt. You are going to stop hurting as you gain more and more distance. Ime, the grief of a marriage ending is a lot like a bereavement. You learn how to live with it, but it is part of you. I try to treat that mourning part of myself with care, love and attention, and if I have a bad day, I forgive myself for it and try to be tender towards myself.
Sending you my love.