My exh was texting his OW the whole time - I knew as I was sitting there with him and the kids when his phone beeped. He claimed that it was because of a busy work project, and also a work colleague who was helping him deal with his mother's death by talking about cancer with him. It wasn't. It was always her. (I discovered all this later when he was refusing to move out, still claiming he was innocent, and I read his emails.)
As time went on, he had it set up so that it would vibrate, flash and beep, and would place it on the table so we'd all see it. I believe that's because he got his kicks out of us not being able to prove it wasn't innocent; I think this as I know from his emails that he asked his OW to a party at our home as a work colleague, and told her that he would find it exciting for her to be there without us knowing who she was.
In the end, what made me read his emails was him brazenly texting his OW while sitting on the settee next to our daughter. She read it and reported it to me, and he made out that she was mistaken, had misunderstood a joke, or was making it up.
Not saying that this is what your guy was doing OP; just that it's amazing what real shits people can be - while you might be slightly sceptical but generally believe that his wife might be cold and uncaring, in fact this might be the mind games going on at the other end. You'd never guess it, obviously, as he's nice to you, and you'd never think that badly of someone.
That's not to say that you should now hide your face in shame or anything. You need to take care of your mental health to make sure that you don't put yourself in this position again. If you're constantly making yourself vulnerable and being hurt, might it be because you don't like yourself very much generally? If so, you need to do something about that.