Is it possible for you to work out why you feel anxious?
Because the options are action are very different between, for example
- You feel anxious because you feel this isn't resolved yet and you are scared you lose him and your children lose a father.
- You feel anxious because you fundamentally don't trust him anymore and are uncomfortable having him around.
The option in scenario 1 is definitely stick around and work on it. In scenario 2 it is work out if your lack of trust is really justified and then decide whether you want to leave or stick around.
It does sound like you have had a bit of a shadow of a marriage since your first child was born. That is a long time to go through the motions.
If I have missed details about his relationship with his female friend, my apologies, but it doesn't sound like she was a real "OW". Is it possible what happened was he felt drawn to some one else, got emotionally close, considered his own feelings over the last 6 years, spoke to you about being unhappy. Then when that possibility was out in the real world he then woke up and saw what he was risking i.e. a marriage, a family, his children's future. I would say how quickly he turned his thoughts around when he saw the other options (B&Bs, flatshares, visiting kids not living with them) is actually a good thing. He came to his senses pretty quickly. So if it was a near miss but no infidelity, he will be very defensive about it, but I don't think it has totally ruled out the possibility of your marriage working.
Agree with PPs who say that as you have three young children and a marriage this isn't something to throw away lightly. Divorce would be an huge emotional and practical strain on you, him and your children. For possibly a couple of decades. Even the most committed divorced dads can do a disappearing act when a new partner appears on the scene and starts demanding either a family of her own or that he is dad to her existing kids first and foremost.
Divorce is something that can be the best option in dire circumstances, but it really is a very difficult option emotionally, practically, financially. If trust has broken down irretrievably, that's one thing, but if you are both is an kind of "afraid of losing the other but don't want to make the right moves in case of rejection" paralysis, then that is sign that there is something there to save.
I mention this as I had a lot of anxiety over being married when I was first married- anxiety that came from actually having something to lose. It took me a lot of work to get past that and I still struggle with it occasionally. I found it easy to think that DH should make more an effort to pursue me emotionally/physically/affectionately and it took me a while to understand that he was just as petrified of rejection as I was. One time he remarked to me something like"No-one tells you how hard marriage is sometimes. When you're growing up it's all hearts and flowers and dream weddings, but no-one tells you that sometimes you need to work really hard on your marriage, like you work on your degree or your job. When you think about it it properly, of course it makes sense that it takes effort, nothing that good comes without you putting the work in, but no-one tells you that. People should tell their kids that, it's make them more prepared. It's totally worth it, but it's not a walk in the park." i think it's especially true that men don;t get much emotional education/discussion growing up about what relationships actually take.
So now if he makes a little move, I make one back and it's like a dance. I know this could be an oversimplification, but I think we are often brought up to think that men are these hugely emotionally robust lumps with limitless reserves of chutzpah and innate unshakeable confidence when often they have the same insecurities and foibles as we do, and sometimes pick up on our insecurities and feel them too.
Would it help to stop going over the pros and cons in your head and just put your efforts into building more pros? I mean this because sometimes action makes decisions easier than analysis.
Anyway, HTH and that your anxiety subsides.