Op ask yourself, if your partner left you tomorrow would you go back to your EXh? Because therein often lies your answer, and/or your motivations.
Affairs are rarely black and white, and more often than not they are symptomatic of something wrong in the marriage. And often they also give people the strength to leave a relationship which they otherwise wouldn't have.
But sometimes if the affair partner leaves for the OM/OW and that relationship breaks down, they suddenly realise what they have lost and seek to go back.
So while you currently feel unable to leave your partner, how would you feel if he left you?
I had an affair. But unlike you I didn't leave for the OM. The affair did however make me realise that I could leave an emotionally abusive marriage, and even when the affair ended and my EXh said that he would do anything for us to make things work again, I realised that the affair had been the symptom not the cause of my marriage breakdown, and I didn't go back. I could have, for months and months I had the opportunity. But I realised that if I'd gone back it would have been for the wrong reasons, because I didn't want to leave my comfortable life with security rather than because I had realised that I loved my EXh and wanted to make my marriage work. So we split, and are both with new partners now.
I can categorically say that the affair is one of my biggest regrets in life, and if I could have gone back and left at a time when I had wanted to I would have. However I can't go back, and so now what is left is for everyone to move forward.
You have to do that too op. You left your marriage for a reason. Not just to be with the OM, but because your EXh had previously cheated on you. He is currently with someone else, but you know that if you gave him the green light he would come back to you. So he still would be prepared to cheat on his new partner to be with you. Is that what you want? Your children are currently hurting, but is that the kind of relationship you want them to see as normal?
Your DD is angry at the moment and feels that you and your H should get back together, but realistically she should be aspiring to relationships where all the parties are equal, and where no-one's feelings are trampled in pursuit of anyone else's happiness.
Now is the time to decide what you want op. If you want to go back to your EXh then make a clean break from the OM and go back. But remember what made you leave him in the first place. And remember that your children are adults, you can only go back for you not for them. Because before too long they will be embarking on their own relationships, and the happy family you see now will be taking on a different form, where you see them maybe every few weeks, months, etc, and you are no longer centre of their world as their world expands in order that they live their own lives.