I think that if a couple are trying to get through an affair, then it will never work if one partner does keep bringing it up. But in order to get through it, you have to talk it all through, have counselling and then put it away. or there really is no future is there?
I am not saying that is what you should do, some people simply couldn't do it and I don't think that I could. My XH was having EA with his best mates wife. I begged him to come back, but in reality, I would have never been able to trust him ever again, because I am a very loyal person and he totally betrayed my trust.
But he is wrong to blame you for ruining the relationship, because he did that all by himself. He can do as much as he likes to put it right, but he cant take away the fact that he did it.
and if you really cannot get past what he did, and I don't blame you for that, then there really is no point in continuing.
If a partner decides to try and make it work with a cheater, then at some point you DO have to forgive and forget, or you will never get past it.