"... he is a fantastic dad with his two kids..."
But you have 5 children. And you need to look at how he is treating your older children. Your 13 year old cannot spend the night with you, their mother, because of your partner. Your 17 year old is miserable, because of your partner. How long do you think it will take before the 13 year old refuses to see you at all? Before your 17 year old moves out and refuses to have anything to do with you at all? Before your first grandchild isn't allowed to spend time with you, at all, and your oldest child goes NC?
Because of your partner.
He might be "a fantastic dad to his own two kids" right now, but that won't always be the case, I'm willing to bet. And he's already turning them against you, too.
Leave him, let him try to fight you for your two children by him - you have already created a history as to why he shouldn't have primary care of them. Your older children can speak up as to what he's like, towards them, you, the younger children, too.
For the sake of your own health, sanity and for those of your poor children... leave him. Sooner rather than later. Because he is a nasty piece of controlling work, I'm afraid.
My ex used to do the talking quietly, then talking over me bit, too. It's a way of controlling the situation by ensuring that we feel frustrated - and raise our voices. Then they can paint us as "unreasonable" and "unstable" and "argumentative" and they "were only trying to say X, but we started yelling about Y...". One day, during mediation concerning our son, he made the mistake of doing it/with the woman who was trying to make him understand that taking a two year old onto a roof, and then leaving them there, was completely unreasonable behaviour and that my concerns for our son's safety were, indeed, perfectly valid... and didn't like the consequences. They trip themselves up/show their true colours to others outside of the relationship, eventually.
But protect all five of your children - and your soon-to-be grandchild, too. Leave him. Let him trip himself up and reveal how appallingly he is and has treated you. 