You are not worthless and foolish.
You have been lied to and manipulated.
He is currently worth less as a person than the one you believed you were with and he's acting like a fool.
Your gut instinct that there is more to it will probably be true, men will always tell the bare minimum of the truth when backed into a corner, just enough to see their way out. I've seen this as both the other woman and the person being cheated on.
You can ask him outright, but he's already lied, chances are he will lie again. You can investigate yourself further, but nothing good comes from pain shopping.
Assume the worst, accept it and decide how to proceed from that. So if seedy nights away at a hotel and a plan to leave you is the worst, just act as if that's what he's done. Make your decisions based on that action. You'll never truly know it all.
You've not been suspicious for no reason, all of your thoughts and feelings are valid. Please do not feel bad for looking at his phone. His actions caused your reaction.
Do you have anyone in real life that you can talk to about this? Just one person you trust that you can be open with. A shoulder to lean on really is invaluable.
He's admitted that they have spoken about being together but that he can't do that while you're together. That means he's considered leaving you. Think carefully about what you actually want here. Do you want to be with a man who was willing to disregard you for someone else?
You are not to blame. If your wife is showing a lack of interest you make moves within your marriage to fix the problem. You don't start boinking the colleague. PLEASE do not accept this trash talk from him. You did not force his hand. He made his own choices. He chose to betray you.
Do not feel bad for him.
Do not trust his excuses or lies.
Most importantly, I'm very sorry you're going through this.
Piss on his toothbrush.