Oh, Op, he sounds like a classic abuser. If he was genuinely a decent bloke who, for some reason, didn't get why rape jokes to a rape victim (or to anyone) are a colossally appalling and unacceptable idea, he would have been absolutely beside himself with remorse, guilt, self-loathing, etc., and desperate to make amends and apologise and ensure you that he'd never do it again. Instead, he got annoyed that you had the audacity to challenge him. His upset is all about the thought that he's been challenged, not about your feelings. He doesn't care about your feelings. He clearly gets a kick out of knowing how cruel and distasteful he's being, and he doesn't want someone to curb his fun. That's how little you mean to him.
If you offended someone accidentally and they told you, would you get annoyed with them for bringing it up, or would you be really embarrassed and contrite and try to make amends? You know that normal, decent people would do the latter, every time.
If you look up signs of abusers online, you'll see that getting annoyed when confronted with their mistakes, etc., instead of being contrite, is one of the classic signs.
You can't make it make sense to you, because you're not an abuser. It only makes sense if you don't see others as real humans like yourself, and if you see them as objects that exist for your amusement/benefit/etc.
Please care more for yourself than this, and leave him. It's only a few months, even with the rape, you should not be having problems like this EVER, never mind in the early days. Your lives are not entangled in any way, and it's too early to be really committed and smitten. LTB, heal yourself, and then find someone worthy of you.