I am not sure if you will see this as I am late to your initial post, however I am hoping I may be able to help give an alternative perspective.
I was the other women four 4.5 years. I was single, he was a married work colleague. I had no children, they had 2 and had been together since secondary school (so around 20 years). We were completely innocent friends for 2 years and there was zero attraction to him on my part. But after he admitted to having feelings for me I realised I too had feelings. Things developed slowly - it was more of an emotional affair to begin with, then stolen kisses and then eventually sex (lots of sex). He always promised to leave her, I always told him to stay (for the children) but as the years went by and I changed my tune he became more inclined to stay through the fear of her stopping him seeing the kids, financial commitments - the usual pathetic excuses. He repeatedly swore on the kids lives he did not love his wife and would do everything to find the strength to leave - or push her to end it by causing arguments all the time.
I am not deluded, I always knew deep down he would never leave and I promise I know what an evil person I was/am (hence my username). I never dreamt in a million years I could do what I did, I thought women who did that were scum - I still do.
I ended it, left for another job, cut all contact and for three years heard nothing until Christmas 2019 when a card came through the door from him. I haven't responded and have no intention of ever doing so.
The reason I am telling you all of this is because I think you need to accept that if he is looking elsewhere he is not happy with you, he cannot love you and you deserve more. If the guy I had an affair with (literally everyone though he was a perfect, innocent family man) can do what he did and then still and make contact 3 years down the line.....I do not believe any man that cheats can ever be trusted again.
It is your life and your choice but please, trust me when I say he will lie to you. He will play it down and no doubt do everything he can to make you think he still loves you. It was a weak moment, he "got caught up in the excitement". It is ALL lies. The trust was shattered the moment he crossed that line with her.