Lifecraft You said the intent was important.
He is telling you his intent was not to be sexist or offend domestic violence victims.
As we have to take his intent into account (as per your stance), then surely he can't be sexist or misogynistic?
There are men who will tell you the joke is not about punching a women in the face, and you have completely misunderstood, that's not what they meant.
I would suggest googling the mental gymnastics of how some men explain alternative meanings to rape and DV jokes, but I wouldn't recommend it - it's a scary place 
To me, I don't care what a man's excuses, explanation, misunderstandings, he uses when he makes a joke about punching a women in the face. It doesn't matter if it didnt enter his head that the women in the joke is suffering domestic violence. The joke will be interpreted as a nasty, sexist joke and he is a sexist, misogynistic arsehole.
In exactly the same way, to me, I don't care what Danny Bakers excuses, explanation, misunderstandings, he uses when he makes a joke about chimpanzees and baby. It doesn't matter if it didn't enter his head that the baby in the joke is mixed race. The joke will be interpreted as a nasty, racist joke and he is a racist.
I know it's hard to be accused of being a racist, sexist, homophobe, islamaphobe, etc, but when we start making excuses as "well, I didn't mean it that way", "oh no, I really meant this", then these issues will continue. As difficut as it is to do, we have to look at our own biases, racial predjucides, superiority over others.