Male colleague: I didn't sleep well last night, it was so hot.
Me: Yeah I was really hot last night too.
Male colleague: Yeah well you shouldn't have been having sex all night then.
I have never mentioned sex in any prior conversation. I didn't really reply but it felt an odd 'joke' to make. AIBU?