I know I have been weak but I have made many (yes too many) allowances to try to understand his need for external affirmation and the fact he hasn't taken any of these occurrences to a physical level.
Putting aside for a minute that I feel certain someone like him has physically cheated, I think it would be good for your mental health to consider the next question.
Do you hand on heart believe that if he had the opportunity - say he was away for a week and an attractive women came onto him, or he got on with someone on a work course and really fancied her - that he wouldn't go through with sleeping with her if she wanted to?
Do you believe that if he thought you wouldn't find out (or that he could convince you it didn't happen) that he wouldn't go through with sleeping with her if she wanted to?
If you think he would then think about what that means for your sense of security and self worth.
I've been in a relationship where there were lies, then lies about lies and constant 'well I couldn't tell you because I knew you'd go mad' arguments.
Once you know they can look you in the eye and lie, or they can say something and know you know it's a lie but you'll stay anyway, it's already over.
Do you think this will get better? Do you think you'll ever be able to not question his behaviour or feel suspicious and anxious when he does something like this?
You will waste your life on this man and he will hurt you over and over and over.
You don't need to wait until you "know" he has physically cheated, or until you have irrefutable evidence.
You know he's horrible to you, a liar and would shag someone else given the opportunity if he thought you wouldn't find out.
You must start making plans to leave this relationship.