I would end the relationship for this 'joke'.
Some might feel that's extreme and I know it's hard for you to hear that when you know the whole person and aren't just basing your judgement of him on this once comment.
It's easy at this stage, and after only one incident to move past it; put it behind you; choose to consider the whole person and all the good stuff that has happened in the past year...
However. It has been said. It can't be unsaid.
I am 50 years old and in my 32 years of adult life, I've been married and divorced, had other relationships, dated and dumped and, yes, overlooked, forgiven and given second chances to men who have made 'ill considered jokes' or distasteful comments at some point and there has not been a single time when I was right to have done that.
What's actually happened is that, by the time I've dumped them, I've got 6 months worth, 2 years worth, how long I've stayed with them since that first comment worth of similar jokes and comments.
There was a thread recently that asked if there are always red flags at the start of the relationship. Opinions were split. But, essentially this is one of those red flag comments that you don't like now but wonder if you're over reacting. In another year or so, there wil have been many more and you'll wonder why you didn't just end it the first time.