I agree that it's not normal to dismiss a finding like this without some kind of curiosity about it - condoms don't just appear in pockets. They either are there becaue the owner put them there or someone else has....and anyone in the latter group would be curious to know how it got there and wonder aloud, especially faced with a gf finding it.
Call him on this. Say you find his reaction odd and unconvincing. Ask him if he isn't at least curious about how it got there...because you are. Move the discussion from one of you accusing him of cheating or suggesting he is, to a conversation about his willingness to engage and discuss how this might make you feel and your concerns. Make any comments about leaving him about this - because minimising this isn't normal behaviour or the signs of a healthy relationship.
I'm sorry OP, but I do think this trying to close down the conversation so you do t even ask the dreaded question,mso he doesn't have to deal with it, is an attempt to control you and stop you asking difficult questions. Perhaps he's thinking up a good story now.
It isn't the sign of someone with an innocent explanation.
This is an early stage relationship even though you have already moved in together. I would try to have the conversation about him needing to talk to you about this rather than dismiss the issue - and if he won't or you feel he is lying because he's had time to dream up a story, simply say this kind of communication and stuff isn't what you need in a relationship, so are walking away. And do it.
I know you want him to be innocent and to keep going. However, you need very strong evidence to do this....and I'm sorry, but you haven't had it and what's happened so far suggests you won't get it.