You sound like you have your head screwed on right.
Something similar happened to me, except one step forwards. I never talk about it in real life, but I have shared it on mumsnet, it helped me at the time when I was lost and scared and didn’t know what to do. And it helped afterwards, when I struggled with being a lone parent with a baby. And I hope it helps now too, now DS is a bit older, I’m still single but don’t miss him at all, and I share parts of my story to help other women. I hope it helps anyway! 
We were arguing and I was asking him to stop spending money and get a job because we had no money to pay the rent (again). I was crying and I reached out to touch him I think, although I don’t remember that. My stbxh said I did so it was my fault because he took it as a sign of violence. So in his reality he was simply defending himself when he hit me. And after all, it was only once so it’s not a big deal right?
I am almost as tall as him, and I was still fat with baby weight so how could he have hurt me, he jeered this at me a lot afterwards, when he realised the attempts at making it go away hadn’t worked. How could he ever hurt such a huge fat lump as me?
Well, my reality was a little different. And as far as I know he still ridicules me and says cruel things because I refuse to agree with his version of events.
But I was and am most unreasonable in persisting with my silly hysterical fantasy, which I’ll share with you, as I’ve spread this ‘gossip’ with others and much maligned his name (& so am definitely the villian in all this).
It was one punch yes. But not ‘just’ one punch, no ‘just’ at all. He punched me in the throat, it came out of nowhere and it was at full strength. My head snapped back and then was banged forwards again by meeting a wall at speed...
The force of this ‘just one blow’ had thrown me through the air and into the wall opposite. I lost consciousness from the force I was thrown into that wall, and the blow to the back of my head. I remember the bang to my head and sliding down, and vaguely being confused as I didn’t understand what I’d hit as I wasn’t aware how far I’d been thrown (about 3meters... but hey, I’m fat and it was just one little hit right?!).
When I came to, he was standing over me shouting at me. I He was shouting at me and saying I was ‘making it up’. I crawled to the phone and then stood between him and the bedroom, where DS was sleeping. Then I told him to get out or I’d call the police.
And he jeered at me, mocked me for shaking and having to hold the wall. Said the police would laugh at me too. I remember I was shaking so badly, and I couldn’t see properly, I’m not sure if that was the fear or the effects of the blow.
He asked me why I was standing outside the bedroom door, and I said it was to protect DS from him. I think he genuinely couldn’t understand why I’d think this was necessary, he was incredulous.
He finally left the flat.
I told my GP a few weeks later, unfortunately after the bruises had faded. But the GP was really nice and supportive. She asked if she could speak to her colleague who had more experience, and then phoned me to say that the single most worrying thing about the incident was that he mocked me and jeered at me when I was hurt, and when I was protecting my baby from him.
In her/ her colleagues experience, the GP said that when an abuser shows no remorse afterwards, and no recognition of the basic effects their violence has caused, he tends to be be very dangerous.
I am left with the after effects of his violence, as my neck never recovered properly, and I can no longer hold my head up for long periods of time, and I can’t go in cars without lying down in a neck brace as my neck is just too weak to cope with even the smoothest of braking, bumps and turns.
So... please continue to be strong. Don’t take risks and don’t think you’ll get gradual warning signs before a fatal or life changing blow.
And take his mockery as the hugest red flag you’ve ever seen.