So OP, how did your DH find out about the affair last time?
You see, the reality is that the reason you are both still together isn't because you wanted to stay with him. It's because he didn't throw you out or end the relationship when he found out you'd been having an affair.
The truth is you're not in control of any of your life right now. You're lusting after a man who goes home and has sex with his wife every night and tells her he loves her, while in the meantime you're going home to a man who is only with you because he made the decision to stay following your last affair with the OM who is now having sex with his wife, you know, the one he married since having the last affair with you.
So the next time you lust after this man and live out your fantasy of your eternal together-ness, follow the fantasy to its conclusion and imagine the way it's all going to end. You know, the bit where your husband finds out, where he ends the marriage, files for divorce on the grounds of adultery, tells the OM's wife, your family, your mutual friends. And where even if they don't cut all ties with you, they will still judge you, somewhere in the backs of their minds they will be looking at you as that woman who shagged another bloke behind her husband's back.
Marriages end, and affairs happen, that is life. However, if you don't want your marriage to end, then the affair can't happen. And if you want your marriage to end, then end it before the affair does happen. Nothing good can come of an affair, ever.
This OM is like a drug. An addiction. While you have contact with him you will always feel that pull towards him. Don't kid yourself that you can be rational about it, you can't. Detachment is the only way. Block his number, don't engage with him, and find another job if you can. In the beginning it will seem pointless. You're friends after all, so why should you block him? But as time goes on you will feel less and less likely to want to be in contact with him. With time so also comes indifference.
I know all these things OP because I've been there. I had an affair which spelled the end of my marriage. In my case the marriage needed to end and it did, for various reasons. But that is irrelevant. The fact is that I had an affair, I compromised my standards, my integrity, my whole persona because I thought that the answer was with someone else who made me feel loved and wanted. the truth is that it was all a bit desperate. I was lonely and isolated and I stupidly believed that my life had greater meaning. It so happens that it did have greater meaning, but not at the cost of my integrity and self respect.
I didn't leave for OM, but he did remain in my life for some time after I left. Promising me friendship, and maybe more in the future, telling me how awesome we were together, if only things could have been different. And looking back the only thing I think is that it was so bloody pathetic. So one night I deleted his number from my phone, removed him from social media and never spoke to him again. And it was like a weight had lifted and I'd been given permission to move forward with my life.
I am with someone else now, and I haven't spoken to OM for over three years. Well technically I haven't spoken to him. My DP and OM know each other on a casual basis, so very occasionally he has e.g. Popped up on social media for instance. but now I am entirely indifferent to him and everything he represented. I have no inclination towards any contact with him. I could in fact sit down with him for a coffee and I know with absolute certainty that there would be no spark, no chemistry, nothing. But as it happens I in fact have no desire to ever encounter him again. He means nothing in my life, and that's as it should be.
If you don't want your marriage to end OP then this affair has to end now. you have the ability to take control of your future here OP, if you don't someone else almost certainly will. Your DH, the OM, the OM's wife, the possibilities are endless, and while you're in the middle you have absolutely no control of where your future is spinning towards.
Don't compromise your self respect for something which will end in nothing.