AIBU to find this comment from a colleague odd?

[KitKat1985]

Male colleague: I didn't sleep well last night, it was so hot.
Me: Yeah I was really hot last night too.
Male colleague: Yeah well you shouldn't have been having sex all night then.

I have never mentioned sex in any prior conversation. I didn't really reply but it felt an odd 'joke' to make. AIBU?

You should have told him at his face that what he said was inappropriate and not something to be said to anyone and to be careful about his choice of words when talking to others.