You say you're not leaving, but you have to OP. This man is abusing you and your children.
Now he's shouting at you and calling you names, blaming you for everything.
Does he take opportunities to "accidentally" hurt you, like bashing into you with his shoulders when you walk past or snatching things aggressively out of your hands?
Does he paint a picture where you are the reason he goes off in a rage?
Does he try and get the children to think it's your fault, too?
Does he deliberately ruin the things you've been doing, like messing up folded laundry or messing up a tidied room?
Does he try to frighten you, for example by slamming things down close to you or driving too fast deliberately?
If he doesn't yet, you bet he will soon. Men like this only get worse. He is testing you now, seeing if you will let him leave or whether you will put up with this dogshit treatment. When you accept it, he will then move to make it even worse.
This man does not love you. Do you understand that? He might say that he does, but he doesn't. If he loved you he wouldn't call you names and frighten you. He knows what he's going. He's choosing to do this.
You're asking what you could have done differently. There is nothing you could have done. Women in this position often try to look for the reason of the anger, thinking if only they behaved differently, it wouldn't happen. It's not true. He knows exactly what he's doing. He's choosing to treat you like this. He's treating you like this because he wants to, not because of anything you've done.
Leave OP!!!! I know it's hard with a baby. Give yourself time if you need to. In the meantime, save what money you can. And write everything down. Start a diary where you careful enter every single instance of aggression towards you. Write it objectively and without emotion. Transcribe what an he said to you precisely , word for word if you can remember. Date it. You need to build a record both for court and for your own self. The fog will come back real soon and it's good to read back and see that those things really happened.