I'm a bit overwhelmed by the answers, I thought everyone would say I was over reacting and being an idiot 
Thanks for all the views. The more I over think about it, the more I think that it's his anger that's holding me back more than anything.
Dragon Of course he has lunch with man friends and man friends on facebook, but he's never lied to me to spend time with men, or had ridiculously inappropriate conversations with men then told me it's my fault so I guess I'm a bit less sensitive about men. I do believe he's an equal opportunity friender, he's not just seeking out women exclusively.
Daisy I'm really sad for your friend's wife that he's lying to her that easily. I can understand it if she is jealous for no reason, that must be horrible for him to live with. But - hypothetically - if he's broken her trust in the past, and every time she has a wobble he yells at her - would you still be happy to be his friend?
supaloops I read your post and nearly cried because I just thought, he must love you so much to help you through it like that.
I completely understand that it could be his guilt messing up his head. But I don't understand how hurting me helps.
He's still quiet and apologetic at the moment and we're both being a bit careful with each other, and if I just keep my mouth shut things will be fine again. When he's calm and ok, he will tell me he loves me, he would never cheat on me, he couldn't live without me, which is lovely of course - but where does all his anger come from? How can he say such horrible things to me when he's angry if he loves me so much? How can he snap so quickly and swing from knowing he messed up to saying I'm overreacting and he did nothing wrong?
I feel like he's really emotionally selfish and immature (and I hate saying that, as I hate saying anything bad about him) because 99% of times when we're arguing, if I say that him doing X hurt me, he'll immediately say "well, you doing Y hurt me much much more!" I'm not allowed to be upset about something - he has to have it worse and it's draining.
And he blames me for pushing him to shout and sadly sometimes (not every time) I do push because, yes, it's a small thing but it's still hurting me, he's still not protecting me, he still doesn't get it. But I also see his side, that it's completely innocent. I just wish he could see my side and understand I don't want him to stop having friends at work, I just want to know he has my back.
When arguments escalate, I'm not allowed to speak to him anymore. And if I do, whether it's me apologising, or asking him to calm down, or getting angry too, if I dare to carry on speaking he'll insult me, tell me shut up, shut doors in my face etc. Obviously then I feel even worse, because as well as the past not getting resolved, I then feel like he hates me so much, so why wouldn't he cheat on me? Then he says sorry the next day, that he didn't mean it and expects me to be happy and fine. I don't think he realises that I have the pain of all the lying and betrayal from the past, plus the pain of every fight and all the horrible words since then. And yet I need to shut up and be fine.
I really feel for him too, because to be doubted must be a horrible feeling. I always tell him that I do trust him not to cheat, but the fights have left me with zero confidence in myself, or that he truly loves me so sometimes I need a bit of reassurance. He honestly thinks I'm abnormal and controlling, yet he's admitted if the situation was reversed he'd be upset too.
I'm trying to keep my mind off it, but I keep having weird palpitations that make it hard to catch my breath, fgs. Why can't I just be cool and confident and not let any of this get to me??