OP before you do anything you need to ask yourself some very honest questions and answer them very honestly.
You say your marriage was already unhappy, was it really? Or have you rewritten history to make yourself believe that your marriage was unhappy? After all, to justify having an affair for three years you would have to convince yourself that what you were doing was the right thing. After all,a three year affair is going to include emotional attachment not just a one off night or two of having something different.
Secondly, why do you want to work things out with your DH? Do you really want to work things out because you love him and see the rest of your life with him? Or is it because the OM is no longr in the picture and you're afraid of going it alone? Be honest, if the OM turned up on your doorstep tonight and told you he'd left his wife and that he wanted to build the rest of his life with you, would you still be thinking that you wanted to save your marriage to DH? Because if not then you're staying with your DH for the wrong reasons. There doesn't need to be someone else waiting for you to end a marriage. Statistically most affairs do not make it past a year in relationship terms anyway so even if you left purely to be with the OM the likelihood is that you wouldn't end up staying together in the long term. There has been far too much deceit in your relationship to be able to start afresh with this OM without a huge amount of distrust and suspicion anyway. And while some relationships do start off the back of marriage breakdowns and do last, on the whole those relationships are relationships where the marriages were ended quickly not dragged out for years while everyone deceived everyone else.
Of course you love him. Unless he's a complete monster you don't go through years of marriage without retaining feelings, even if those aren't necessarily romantic feelings. From my perspective I loved my H but he was emotionally abusive, and the marriage was in trouble long before I had an affair. In fact we had talked about splitting up even before I had an affair but didn't.
But when I had the affair I thought that the OM was the greatest thing ever, that he loved me, that I was the best thing to have ever happened to him and vice versa. Affair was brief though, but it did make me see how unhappy I had been and how I couldn't stay where I was. But when the affair ended I too thought that I could just stay in the marriage and hope for the best and maybe rebuild things for the sake of the marriage and for the children. But then I thought about everything that went before, and I realised that my wish to stay had more to do with the fear of leaving with nothing to go to rather than because I wanted to spend the rest of my life in a marriage which, although it had its good points also had abusive ones. So I left anyway. And the night before I left I thought about what I was doing and wondered whether I should in fact stay. He would have taken me back at that stage. But I didn't. And now several years down the line I know that even if I'd stayed things would have settled for a while and then gone back to the way they had been before and we would have been back to square1, so I know that leaving was ultimately the right thing to do.
As for the OM, he did in fact try to come back. But he'd subsequently got together with someone else and thought that now that I was free I might like to be his bit on the side while he decided whether to be with me or her. I told him where to go and we haven't spoken or had any contact for years.
Ultimately the future is uncertain. But remember that your children won't be children for ever. One day they will grow up, leave home and lead lives of their own. And if they are the reason you feel you should work on your marriage with your h then you need to ask yourself whether you can see yourself loving him for the rest of your lives even when the children have left and it's just the two of you, or whether you are just doing this as a temporary fix because you see your whole life as one with small children when in fact it isn't going to be like that.
The OM is gone. The relationship with him is over. Cut him off, he's made it clear what he wants, it isn't you and you're not in a position to change that. Whatever you do now you have to do for yourself. Not for your children, not for your husband, for you.