So, preparing to be flamed here but I don't see this as sexism. I mean, it COULD be, but it's not necessarily sexist.
It's a bloke either having clumsy banter after the OP joked herself that she tries not to work unless she has to, or a bit of a sleazy flirt. I'd lean towards the latter. But being a sleazy flirt isn't sexist - it's being a sleazy flirt.
I've seen plenty of women make inappropriate comments to men over the years particularly in offices where the bloke is outnumbered. That's not saying it makes it right but I don't think it's an issue confined to women and I think it's creepy and unwelcome, but not sexist.
If it's the first time, I'd have probably had a bit of banter back - but that depends on his personality and how well you get on. I'd have likely said something daft along the lines of "Well, yeah, I tried working street corners but the hours weren't great" but that's because I tend to use humour to defuse situations.
If he said it in a really obviously slimy way, or he had a history of inappropriate comments, I'd have made a crack about his wife.
I mean, he sounds like a dickhead and a bit of a pervy lech, but unless he's demeaning how well you work or it was a persistent problem, I'm not sure how much I'd care.