Part of that imho is that their victim thinks people tell the truth, that their partner must tell the truth .....
Their partner appears so sincere/convinced/earnest/adamant that this is what happened (my exes were violent and I got caught up in it, and the police believed their lies that it was only me who was violent", "you wound me up, you set me off, you wouldn't do Abc that I asked, you did ABC that I told you not to do") and in a continuation of our naive automatic childhood assumption that people tell the truth (especially people we're bonded with/intimate with etc.) we think they must be telling the truth.. They wouldn't lie about that, would they? How could they?
Well, people lie all the time.
Just because anyone says anything, no matter how convincingly, doesn't make it true.
People lie when it's in their interest.
People actually convince themselve that their lies are the truth; to the point where they're very convincing.
He's lying.
He's lying about his exes, and he's lying about you being to blame for his choice to be violent.
He would choose a decidedly different path to express his annoyance etc. if he was dealing with a big bloke who could beat him down.
(And I bet what he's getting annoyed about is completely unreasonable and unfair too, from your op).
At some point you have to gain some very healthy skepticism.
It's long past time for you to gain that skepticism op; to see his past for what it is, to see him for what he is, and to free yourself and open a new chapter of your life.
Otherwise you're going to get kicked around, battered, hurt, mistreated, and abused - while you deal with a chronic illness - for the foreseeable future.
That's not even getting on to the other types of abuse he's been subjecting you to.