I was in the same situation nearly 5 years ago now. Husband left me finally (after much coming and going - although it was me that had the affair - long story, but decided to give marriage another go - had another baby, he then changed his mind). He left me when my youngest was 11 weeks old. My other two were 4 (nearly 5) and 9 at the time. Such bad memories when I think back but now we have come so far.
It all came to a head when I went on holiday with my girl friends and our children but left my 9 month old at home with (ex) husband as all the other children were roughly the same age. I found out whilst in Spain that he was spending the bank holiday weekend with his girlfriend and my baby. I know she was only 9 months old but the thought of this other woman holding and cuddling my baby when I was missing her so much was unbearable. He moved out the day I came back off holiday - lived in the loft for the 9 months after he had left me (just couldn't get him out of the house).
Now, okay, it is nearly 5 years on - but we all get on so well. Ex has children every week over night at his girlfriends house (where he now lives) and he actually has them at the moment for 2 weeks (as I work and have childcare difficulties in the summer holidays).
My boyfriend can meet my ex-husband to collect the children or I will quite happily drop the children to his girlfriend.
It is such a hard situation to be in, especially as, for you, it is still so "new" but take it from me, someone that has been there and it is just such a "shitty" time, you will get through it, you will be fine, the kids will be fine. If you explain how you feel, I am sure that he only wants to do the best for his children as well and will totally understand where you are coming from. It is hard enough for the children to accept that their mum and dad are no longer together but they do not need to see the "replacement" until he is sure it is long term and then it has to be a gradual process.
My children totally accept my boyfriend and also their dad's girlfriend. My eldest (now nearly 14) tells me that she gets upset that me and my ex-h are no longer together but she can see that both him and his girlfriend, and me and my boyfriend, are so happy together.