So this happened right at the beginning of your relationship? You didn't get married for 6 years after it happened. And he chose to marry you after it happened. He had more than enough opportunities to decide not to pursue a relationship and marriage with you if he felt he couldn't trust you. Let alone 30 years since marriage. Affairs take, on average, 2-5 years to recover from. Sure, there are outliers, but this is so far beyond being an outlier.
For context, I have been cheated on. I have forgiven and remained married. I have never brought up his affair in a fight. That was one of the conditions we put on deciding to reconcile, that while we could talk about it if I wanted, I would never use it against him. No one should live with that threat hanging over them.
I would seriously consider leaving him at this point, and certainly would be putting a boundary in place with him that if he ever weaponises it again, that you will leave (as long as you are prepared to follow it through).
I feel so sad and angry on your behalf that this man is being so cruel and has broken you down like this. I would be willing to guess that this is not the only problem in your marriage, that he is controlling and domineering in other ways, and uses your guilt to keep you 'in your place'?
I wish I could give you a hug.