@tequilasunrises YANBU obviously.
I have experienced this kind of person quite a few times in my worklife!
I used to have a manager who would comment EVERY TIME I got up to leave for the day. We had flexi time, and could do our hours anytime between 7.45am and 6.15pm. (We had to do 7.5 hours, with a half hour, 45 minute, or one hour lunch.)
So I would usually do 8am til 4.pm and then dash off as it was the end of my work day, and I had things to do (being a working mother and all!)
At every available opportunity, my manager would say 'SOMEone is in a rush' or 'where's the fire?' Or he would say 'hey you lot, an impression of Lavendar,' and he would shoot out of the office, and people would laugh. Really fucking pissed me off.
Most of them would come in for 9 to quarter past and leave at 5 to half past five. OR they would do 8 til 6 for 4 days, and then finish at lunchtime on the Friday. The comments from the smug cunt of a manager really pissed me off so much.
When I challenged him on it, he said 'I just love to make you squirm, and see you go red.'
I said 'I am doing my hours you know!' He said 'yeah but I love to see you go red! Your neck and everything goes flame red, it's hilarious. You're so easy to wind up.'
He didn't stop.
Cunt.
Even when I had a quiet 'unofficial' word with someone in HR, I was told to not raise a complaint about something so trivial, as it would make me look like a silly child.' 
I was glad to leave that job! I put up with his cunty comments for 3 years.'
@regmover
Or you could say something like "I don't know why you comment on my hours at least twice a day, but it's starting to feel like workplace bullying so please stop."
I would say this. ^
But add onto it 'if you don't stop, I will report you to HR.'
My Manager problem was some 2 decades ago (1999 to 2001 ish.) At the time, as I said earlier, I was dismissed when I mentioned it to HR, and told to stop being silly. I don't think my complaints would be dismissed these days, and he would have been told to stop.
I have no idea why people say this (as there does seem to be one in every workplace,) but it's fucking annoying. Are they jealous? Are they threatened? Are they just fucking goady? I don't know why they do it.
Maybe someone who DOES say this kind of thing can enlighten us all???