The first comment said it all....
Just tell him straight up: "your jokes are not funny and I find them insulting and offensive. Stop making remarks about me or I will take this further"
Some people think they are funny (especially men!)
And silly little petty remarks are just 'banter,' but it can be very tiresome.
You need to tell him straight that the comments are pissing you off. Say, 'I have a good sense of humour, but the same thing being repeated over and over is just boring and annoying. I am entitled to come what time I like and leave what time I like. Do you not get the concept of flexi-time?'
I worked part time - 3 days a week - when my kids were little. I started at 8am, and finished at 2.45pm. (6 hours 15 minutes a day, with a 30 minute lunch, 3 days a week.) My lovely hubby took the kids to school and I would get them back.
Anyhoo, I used to leave at 2.45pm, and every fucking day, this one bloke in the office would make a comment.
He said 'oooh all right for some, working part time, part-timer, slacker ha ha lololollll!! 
Fuck off!!! 
He was only joking though.. ha ha ha lol rofl lmao! Arsehole.
He was 28-ish and lived with his parents, and there is nothing wrong with that, but the fact was that he did 9 to 5, and then went home and played on his xbox for 2 hours, had his dinner then went out with his mates, or sat in watching tv. No responsibilities, no bills, no kids, nothing!
When I left the workplace at 2.45pm, my 'work' did not stop. I picked up my kids, then popped to the shops for some bits, went home and played with them for an hour, did some ironing, hoovering, washing, preparing dinner and so on. I didn't mind, as I love my family, and am happy to look after them, but don't act like the ONLY work I do is the 18 and three quarter hours a week that I'm in the office, you asswipe!