My DH considered leaving me (and our DD) for another woman he became close to when he was depressed. To this day I'm still not certain how far they went with each other, at the time he said it was an emotional affair as such but I think they kissed, and if there was more I don't think I will ever know. I can't imagine a woman expecting a man to leave his wife and child if there had only been meaningful glances between them etc.
It was a horrible time, the most unhappy I have ever been in my life and I was in a very dark place. Like you my imagination was on over-time wondering about them together, what they had been up to, what she had been to him that only I should have been. It was utterly miserable. It came up constantly between us, it was emotionally exhausting and I was a wreck.
Then one day I just decided I had literally had enough of thinking about it - it was literally that sudden. I just couldn't anymore. It was too draining. I stopped bringing it up with my husband, and gradually over time I thought about it less and less. It was like every time it got dragged up it opened the wound and I just couldn't live like that any more. The less we dwelled on it the better our relationship became over time, until eventually we got back on track and were happy, and I regained my trust in him. It was literally a case of 'least said, soonest mended', because actually there comes a point where there isn't really anything more you can say about it, you can't pick it apart any further, it has happened and it is now time to move forward. But I think you have to get to that place naturally in your head, you can't force it.
We are now 6 years on from that time and a few months ago my DH became depressed again and is contemplating the possibility of moving out for a while to get some head space. I know there is nobody else involved this time (I would know the warning signs as I've been there before). God knows what will happen and whether we will get through this rough patch. But the optimist in me does believe that if you are meant to be with somebody then you will find a way through, and if you aren't then you have to try and accept that a different path awaits, as scary and horrible a prospect as that is.
If you are confident that you can trust him from now on then it is worth getting through this, and your new-found trust will get you through the hard times. If you can't trust him then you need to have a think about whether you can live like this going forward.