SaintJoan perhaps I am missing something here, but your further post says that in relation to the affair "he wouldn't give you details....and you reconciled" and later that, given your family background and dislike of secrets, you're the sort of person who "wants to know everything." But it seems you reconciled without being told anything much at all. That's the bargaining I'm referring to.
As you will see from my posts on here, I very much think that marriages can recover and get stronger after an affair, but only if the whole story is "out there" and the couple understand the meaning of the affair; why it happened and crucially, why it can never happen again. If this doesn't happen, it is very likely that another affair will happen, given an opportunity, because nothing has been learnt about fidelity.
The problem too for your H is that after a first infidelity, the taboo was perhaps broken and having never properly acknowledged the hurt caused by his infidelity, he never had to live with the consequences of that broken trust. So perhaps he just got more careful about later dalliances, bargaining that what you didn't know, couldn't hurt you and as long as this time these relationships were no threat to the marriage, he wasn't doing anything much wrong.
The reality however is that when someone's behaving that way, they aren't truly giving themselves to the primary relationship. There is a distance there and like I said upthread, secrets really have an amazing impact on intimacy.
I think what you and LadyButterfly are seeing is that if a first infidelity is not
dealt with properly, by the unfaithful partner or the person they've betrayed, it will usually happen again.
Please also understand if you feel my words upthread were too harsh, that the person who has done wrong here is your H. His behaviour started this cycle and has consequently short-changed you out of the marriage you could have had. But reconciling without dealing with the issues was also a choice you made and with hindsight, it was possibly the wrong thing to do.
You can't "un-know" this information now and so I really do think you should speak to him and ask for complete honesty now. I think you can get past it and build an even stronger marriage, but only if your H is willing to put in the work at analysing his own character and facing up to how he has hurt you over many years. The most uncomfortable truth he perhaps needs to acknowledge is that just because you didn't "know", he was still hurting you. In a myriad of subtle ways, when he wasn't "there", making an active and present contribution to the marriage. And the sad thing is, he short-changed himself in the process.
I think you're both going to need some help with this and would suggest some counselling, but choose carefully and pick someone who understands infidelity and the bargains people make.