Obviously if he meant it or was doing it to make me uncomfortable, that would be very different, but when it's a joke, isn't that banter and therefore fun?
I quite often set the lines up for my colleagues to knock down - it's great seeing them so pleased with themselves for coming up with a joke I wrote for them! Makes office life far more fun too.
But, I am absolutely confident that I have their respect on a professional basis and I think the fact that I can deal with the banter adds to that respect TBH.