Op I can really understand your predicament!
I had a similar situation, there were 2 instances with my ex husband (you'll note the EX!) one at the beginning of our relationship he disappear for one night, said he was tired and at home but when I went over the next day he'd left a train ticket on the side that was him returning from a nearby city that morning, he then changed story and said he with friends for a second night out in a row but didn't want to tell me, but it never sat well with me because when I then said I'd like to speak to his friend to ask he went and shut himself in the loo and text his friend "if she calls tell her I was with you!" !! (we'd only been together a year at this point and I should have listening to my gut and left).
Fast forward 5 years from then and we're engaged and he makes a friend out of this woman in another country where he was for work, involved an emotional affair at least, everything trust wise was always amplified because of that first instance when we were first together.
Fast forward another 5 years, we're married and have 2 children and trust issues still niggled because of that bloody night 10 years before, I put it aside for years but other little things popped up over the years that niggled, and I never felt I could rest easy because the trust wasn't there. It got to the point where other problems we had in our marriage didn't seem workable to me because of the trust issues that stemmed from that first instance and I left him.
What transpired since is that he actually had gone to meet a woman that night even though he swore blindly to me for 10 years that wasn't the case. Also transpired that he just is generally a huge liar and a controlling person.
It's been a tough few years but now I feel so free everyday that I don't live in a life of constant doubt and mistrust. It's a great feeling. I have a wonderful partner who I trust implicitly and is a million miles away from ex h.
My advice is you either need to find a way to put it behind you for good, well and truly, move on and never think about it again (as much as possible) and do what you can to live your lives together happily and free of doubt, or leave, if be prepared to live a life of doubt niggling away at you.