Oh Orm, I really really feel for you. You've had a terrible time of it and I've been through many things similarly, on both sides I'm ashamed to say.
I have absolutely no judgement for you at all and you mustn't think anyone knows better than you about your own life, of course they don't.
The most awful destabilising thing about an affair is an inability to trust again, and particularly to trust your own judgement even. So when that's attacked from any area all over again it's the ultimate pain and humiliation.
What's horrifically painful is also a recognition that one may never know the truth unless the cheater chooses to bare all and the cheater is always in a position of total fear, instability and shame. To minimise what they've done isn't just what they might do, it's what they ALWAYS do. With almost no exception. It makes them feel safe, in control and less ashamed. It's a stage to be got through. Without proof, of course you are stranded and emotionally wrecked with nothing to cling on to. The burden seems to be on you to prove they HAVE rather than him that they HAVEN'T. And since no one but them can truly know.....
But wasn't there a rumour that they were caught at it? Why would that one rumour be made up and untrue?
Look of course there are people who get emotionally close and don't have sex. But I don't think this is the case here. What do I know. Sure. But you can't rebuild a relationship of sand. He has to tell you EVERYTHING or nothing can start again.
Generally recent studies suggest around 85% of male infidelity is never discovered. Think about what the likelihood that this man has fallen in love with someone else and not had sex with them and never been unfaithful with anyone else is and now it's all been discovered and you know it all. Describing 20 years with no lies and 6 months as lies sounds unrealistic. Everyone lies. What he's lies about and when is far more complicated than you can hope to understand right now and you shouldn't feel guilty about that, he should.
I would think very hard about past events, snoop, look at old diaries, receipts, but at the very very least he would have to attend counselling and you should speak to everyone you know who might know something. Often people are desperate to talk in these scenarios but don't feel the spouse wants to hear it.
If you just feel too raw and hurt and like any more information will kill you, that's totally understandable. But you must not look to him for kindness and reassurance at this point. He is the one poisoning you, he cannot be your antidote. It's incredibly difficult after a long marriage but he is the one who has completely broken any trust. You can not be expected to trust him automatically over ANYTHING anymore. And that is his doing. Assuming he is lying is NORMAL. It is fine. It is what his actions have brought about. It is the proper way to respond. The only way he can win that trust back is by telling you everything, cutting her out of his whole life and working tirelessly to win you back. Without that, and you will know soon enough, you will never be safe in this relationship again.