Hi OP, I wasn't sure if I should post or not as wasn't sure it would help...but it might, so here goes.
Twenty years ago, I was married with young dcs, but I fell in love with someone who was also married with young dcs. It was the most gut-churning heart-wrenching thing I've ever experienced. I loved my DH and respected him. I could never have actually slept with OM because of that. And my own mentality would never have coped with sleeping with one man while being married to another, still wouldn't these days. I'm not judging you, just saying I'm different.
But my feelings were so powerful for OM that they took over my life. About six weeks of this torture nearly undid me, and I was unable to live and behave normally so I told my dh, as he knew something wasn't right. He was gutted.
Because I'd broken down crying that morning, DH had said we will talk in the evening. So during the day, I told OM I was going to tell DH everything that evening. He said he should tell his dw too 
DH and I talked for about four hours that night, which ended with me leaving to stay with a friend for the night, at dh's request.
I went and looked at my sleeping dcs before I left 


At the friends house I totally lost it, devastated at what I'd done. No sleep. In the morning OM rang me at friend's to see if I'd done it and if I was ok. (I'd said I would probably end up staying there) I broke down and told him it just felt so wrong...that I was going to ring DH and see if he'd let me come back to talk. OM said he too had told his DW, she chucked him out and he had spent the night in his car.
I ended up going back to my DH and leaving OM totally in the lurch. DH said I could go and meet OM that day to tell him it was over and that I was staying with my DH. He said but don't be too long. Must have been agony for him.
For both of them it was agony, for me it was agony, the guilt at what I was doing to both men was mind numbing and made me ill.
I went back to DH, tried hard, went to relate, but eight months on, we ended our marriage. I had ruined it for both of us. And I still loved OM. Split without knowing if I would end up with OM.
Because of my guilt, cowardice and indecision I put all of us through hell for eight long months. And my DH wasn't, thank goodness, having a breakdown. If he had have been, I may well have stayed, as I couldn't have lived with myself. (Thinking of your OM's situation)
DH2 (OM) and I are still very happy together 20 years later. We have been together longer now than with our first spouses. But it nearly didn't happen.
I realise it is a bit different to your situation, but when I failed to leave because I just couldn't bring myself to, my OM never pressured me. I would say right that's it I can't see you or speak to you, I'm staying with my DH etc, he was gutted, but he let me go. He loved me enough that he truly wanted me to do whatever I thought was best for me.
Your OM can't do this. For his dcs, for his dw's mental health, he just can't. Maybe he thought he could, but when it comes down to it he can't bring himself to hurt them all over again. Respect that about him, however much it hurts. You need to be honest now and sort out your own marriage. That's all you can do.
I'm sorry you're going through this dreadfully painful experience. And I cannot even imagine what three years of this has done to you all. It is time to repair yourself. That's actually all you can do. As long as OM is with his wife you need to not be in contact with him. Or you will suffer even more. If he is decent he won't see you now anyway, if he truly wants things to work with his dw. If you truly love him you need to wish him well and let him do what he needs to do, agonising though that is.