I think that you should just tolerate this feeling that you have about your new partner,and his taking a holiday with his ex and their children.If the holiday was booked earlier,then his children will be looking forward to it.
All this talk about him getting back together with the ex and having sex with her is just an idiot way of thinking.
When I was divorced many many years ago,my ex and I,and our two teenage children,went on a previously arranged holiday to France,on a boating holiday on a river there. We had been divorced for about three months,and my ex had already moved in with a new partner.
We had a great time,and managed to have a happy holiday all together.
The other woman was highly disgruntled about all this, but basically I couldn't have cared less. My ex and I did not sleep together on the boat,as the sleeping arrangements were very good on this large boat.
My ex eventually married this new partner, who was about 20 years younger than him and me,she was a bit of a control freak,and insecure I think, and not mature about things as myself and my ex were.
So I think it's quite ok for your partner to take a pre arranged holiday with his ex and their children,without any odd ball thoughts about everything. I think that you already stated,that he wouldn't be doing this again in the future.