I think to give you courage you need to speak to a RL friend about this. Shine some real daylight on what you're doing. At the moment, you've been having these lovely butterfly-in-your-stomach feelings about how amazing he is, he's your soulmate bla bla - so far, so teenage and frankly so divorced from reality. Today you've come on an internet forum for some input from faceless people who don't know you, or him, or your family - still divorced from reality.
My husband left me for an OW and he didn't talk to anyone about doing so but her. I wish he'd had one single, solitary buddy that he could have gone for a beer with who could have said 'what the fuck are you doing? Are you mad?'
In the pressure cooker environment of an affair, with rose coloured specs glued on and no sense of consequences or what happens in your ever after when you've wandered off into the sunset, there is no perspective and no real vision of what you're doing. At all.
I'll tell you what life after going off with your OM is like. You hurriedly find a rented property near to where one of you lived (and then can't get away from the vengeful ex-spouse) and, in your situation, you live hundreds of miles away from where the other one lived. The house you rent needs furnishing and equipping, so you spunk a load of cash on crappy furniture to tide you over. You pine for your children and have to make do with seeing them a couple of times a week (if you live in your old neighbourhood) or maybe once a week if you moved to where he is. All your time with your children is rushed and doesn't feel like enough. You have to see look of hurt and bewilderment and anger on your ex's face, and try to pretend for the children that everything's basically fine. When you see them you realise that they are doing things that you're not part of any more, because you moved out.
Meanwhile he is also trying to fit in when he will see his children. When you take out all the evenings and weekends when one or other of you is having contact, you don't have a lot of time left together in your romantic idyll. When you are together, no doubt you have a lot of sex in your rented bed but the rest of the time you are trying to get used to each other's ways of doing things. And maybe he's not so perfect when you sit on the toilet seat that he dripped on, or he sits there moping about his children and wondering if he did the right thing.
Your parents, your husband's parents and all your friends look at you and wonder how on earth they judged you so wrong. They wonder how you could do that to your children and hypothesise about how long the new relationship will last.
You have less money than before, and so does he, because you are now maintaining three homes rather than two. You have grief from both spouses, especially the wife, who wants to take him to the cleaners financially.
That's really just for starters. For goodness sake shine a bright light onto what you're doing and stop talking like some drippy 14 year old.