To be fair, it would be a worse blow to your children's life to stay with an angry man. In turn, they, thinking it normal, will go in to have abusive relationships also.
It might not come to it but, you need to be prepared to go if it does.
The 'he can make it seem like it's my fault' is another red flag worry often found in abusive relationships. They make you feel like 'everyone will think xyz' about you. They trap you in a cycle of 'prove your goodness/worth/innocence/loyalty/love etc...' and you get so busy doing that that you don't stop to realise it's all a trap designed to keep you from escaping.
Frankly, who cares what other people think! You don't owe anyone a relationship. And if you leave him, most people would assume 'she must have jolly good reason as women don't tend to leave men when they've got 3 kids to think of unless they have good reasons'. Which, you have, arguably already. Certainly if the behaviour repeats.
Nothing is worse for a child than a household with mum walking on eggshells trying to placate a bad tempered bully. Seeing their mother sad and ill treated.
And let's be real, even when it's a 50 50 situation childcare wise, is he actually capable of that? Lots of guys may say they want that...but won't be arsed to do it.
It's also a common concern that you'd have less time with the kids but, many women then find that actually it's a good thing for them as it gives them time for other pursuits and to have a life outwith just being mum. It just seems a scary thought at first. But kids are better off between two stable homes than trapped in one unstable home.