I’m sorry you’re having to deal with this. It’s always a decision only you can make, all I can say is that in your place I would end the relationship now.
Partly because, as many others have noted, a relationship where he’s being physically abusive has to start with the first action of aggression, and very often it’s exactly this - where he’s ‘moody’ and doesn’t mean to harm you but just does something thoughtlessly heavy-handed, oh maybe just an unthinking shove but you go flying etc. Then he’s utterly ashamed and apologetic. But how many women can say yeah, been there once and he apologised and it clearly shocked him & he never, NEVER did anything like that again? In my experience men can control their physical reaction to their own moods, or they can’t. And the ones who can’t, pretty much never learn how to.
For you, also, I think your own past reduces the chances of things getting better over time now. Your experience makes you very rightly apprehensive and alert to any repeat of your bad past relationship. Now this guy has hurt you, no matter how much it might be a sincere mistake and you both want to put it behind you - to be blunt, will you be able to? Or will you now always be on eggshells, in case you do something perfectly reasonable but he makes it his excuse, again, to get moody and volatile? Do you honestly think you’ll really ever be comfortable around him again? I know I wouldn’t. I’m sorry I can’t tell you something more cheerful or optimistic, OP, but from my own experience I feel uneasy for you if you stay with him.