Lucy. You are in shock at the moment and may not take these words in, but you have my huge sympathy. When something like this happens, it feels as though the world has stopped turning on its axis.
To clarify then, he has been having an affair, only told you about it when you were going to find out about by other means, exposed you to a violent scene and criminal damage of your home and says it was partly your fault? And is providing conditions for staying with you i.e. he will only do so if you forgive and forget?
It is entirely possible to recover from an affair and build a stronger marriage, but not with your H's mindset.
You are quite right. The affair was not your fault at all. Both of you shared equal responsibility for the marriage and therefore equal responsibility for your actions when things got tough. Your husband had a range of behaviour choices to fix your marriage if he felt unhappy; having an affair was by far the worst choice and don't you dare take responsibility for that. That was his choice and his alone.
He needs to take full responsibility for his actions and this will only work in the future if he is genuinely sorry, never once blames you for his choices and gets to the bottom of his own character and why he chose to do something he knew was fundamentally wrong.
Please don't allow him to do a number on you - in your shoes I would be telling him the marriage is over and then watch and wait for how he behaves. This man needs a reality check.
There is lots of support on here for you too - so do use it.