Oh dear OP, poor you. How horrible. Please do tell your mum the truth and get support. Please do call WA.
He's hit you and it's escalated. It will escalate further, it always does. The man will get more and more extreme as you normalise and excuse and let him get away with it. You're already making excuses - he was angry, it wasn't in front of the children etc. There's no excuse.
No, you shouldn't go back, you should keep the hell away from him. Tell your mother what he's done and get support from WA or similar. Read about domestic violence and how it will often lead to murder, how it always escalates. How it will impact your children if they see it and normalise it. Do you really want them to see their mother like this?
He isn't a good father - a good father wouldn't hit the mother. Ever. And walking away was and is the right thing.
My mum was the child of such a house. It's damaged her so much. my ex dp too. He's shared so much with me, including the damage to his sisters, and I wouldn't wish that on anyone. Even the fear of violence, the control, the walking on eggshells, damages the children.