He's saying I'm turning round what he was saying yesterday and this is why he gets so frustrated with me
This is him practising DARVO
"Deny, Attack, and Reverse Victim and Offender."
I'm going on like a dv victim
Well... if the cap fits! (Him not you)
Flicking, pinching, "playful" slaps (that may not even physically hurt at first), "accidental" shoves etc are often the beginning of dv, it's a gradual breaking of the boundaries and your bodily autonomy I've even seen and read things like overly firm hand holding (my dad would do this when in public and he couldn't shout at or otherwise abuse the family member "disobeying" and "embarrassing" him. My ex did it as a reassuring thing before he knew of my history with this and was horrified when I told him. He comes from a very gentle, loving family)
Not all dv results in a bruise, often these bastards are very very careful to leave no evidence - especially if they have a history of being like this.
You say he's in his late 30's, I'd bet good money you are far from his first victim.
How has he described his previous partners/girlfriends?
To be honest op, with your own history I think as soon as you knew he came from a home where there was dv and there was even the HINT of aggression from him you should have ended it. Yes we can say hindsight is perfect vision but that would have been 2 major red flags.
I suspect you know what things are red flags, I suspect the problem is you tend to err greatly towards ignoring them - that's what you need to work on.
But for now the priority is to get him gone as safely for you as possible. Don't react in knee jerk fashion as that could put you at risk. Speak to women's aid and take their advice.
Kindly meant, but it seems clear to me you aren't best placed to make such plans yourself.
I hope you can get an annulment but I think it is unlikely as it doesn't seem to meet any of the conditions unless not consummated which you haven't mentioned (and needn't).
So you may well be looking at having to wait another 3 months and getting divorced.
With such a short marriage he is unlikely to have claim to any assets but you should check with a lawyer.
As I say seek advice.
As to why? Because they consider themselves to be weak, insignificant, powerless people at heart and seek to feel strong, important and powerful by tearing down another.