I have been on the other side. I was the other woman.
I'm not proud of it, and I deeply regret hurting the people that we did. I thought it might help you see a different perspective, and I'm sorry if it causes any further upset.
I was in a 'relationship' (or whatever you choose to call it) for around 15 months-ish with a man who wasn't married but did have a long term partner and children. I was single. He repeatedly told me he would leave but never did. I was a fool for waiting for so long. So I ended it. He then told his partner about us and left. Within a few days he went back to her, telling me he couldn't bare to be apart from his children. To me, it didn't matter the reason.
He continued to message me every single day, despite apparently rebuilding his relationship with her. At first it was apologies, then he told me he didn't feel anything for me anymore and was focussing on his relationship with his partner, then a few days later he told me he had 'bury' his feelings, regardless of what he wanted. By December he was back to declarations of love. I continued to tell him we had no future as he'd gone back to her, he essentially made his bed, and now had to lie in it.
He then told her in December that he had been messaging me the entire time and couldn't continue to live that way. He moved out of their home.
The point I am trying to make is that, whilst I appreciate a lot of men have truly made a mistake by cheating and do regret it, but that the man I knew told the mother of his children to her face that he was in love with her, had made a terrible mistake (which may be true) but was still contacting me. I think you need to look really closely at his actions and how committed he is to rebuilding a life with you. It's been nearly 2 years since all of this began for me, and I cannot imagine the pain we caused his partner. Nobody 'won' in this situation, as far as I am aware they're not together, I'm single, and who knows with him.
I would really urge you to think about whether his actions show how truly sorry he is. My situation is probably different due to the length of time we cheated. But put you first, ignore what he wants, what is it you truly want? If it's a relationship where you don't have to flinch when his phone goes, or if he's late from home, then I don't think this man is for you anymore. But only you can make that decision.