I don't think carrying a condom is necessarily worrying. I had a bf years ago who always carried a condom in his wallet. It was habit and the same one was still there a year later. I wasn't worried about it.
But the fact that he was defensive is dodgier and the fact that he's taken it out, and the fact that he's out a lot more than usual with workmates.
Keeping a close eye on him... well you can if you like.
But is someone you have to police really worth having?
I can't believe the number of women on here who keep close tabs on their man, when really if you have to do that the 'relationship' is a waste of time, isn't it?
You can't make someone want to be with you only, or stop them from transgressing if they want to do that.
I'm not unsympathetic, it must feel horrible. In your position I'd get rid. But I've just left my husband who wasn't even unfaithful and I don't really want to be with anyone right now, so I'm not best placed to identify with anyone who wants to hang on to a relationship.