He says you're abusing him for talking about the affair for four months, four months is nothing.
In my experience he'd be lucky if it wasn't still being talked about in four years. (Personal experience, dh had a couple of EAs about five years ago, 'relapsed' about three years ago. I am not a violent person but I could have ripped him limb from limb at the very least, then put him through worse physical and emotional pain again and again. Affairs are one of the most hurtful things to go through.)
He is not a good husband for putting up with you talking about it. If he was anything like a good husband he would, apart from not having done it in the first place, own up to his mistakes and grovel at your feet for even thinking about thinking about having an EA.
I thought about thinking about getting revenge on my dh for what he did in the past and just the thought of thinking about the possibility of having an affair, emotional or physical, repulsed me, never mind properly entertaining the possibility of doing it.
He's going to try putting a lot of blame on you. Imho he is utter scum. To even consider getting through this kind of thing with a marriage intact takes months and actually working through it takes years. But just from what little you have said it doesn't sound like your husband will face up to it and take responsibility for his actions. Please be very careful in what you believe that comes from his lips.