Ooooo! I wish your husband could do that and make a short arse comment to the chap that's being inconsiderate at best, to the op.
I would actually pay to witness his face when he realises that he's on the receiving end of his own words.
Size should be neither here nor there.
There are exceptions, like when you go to some rides at a funfair, if you are not as tall as the pre measured line on the board, then you aren't permitted to board...
.... but obviously that's a safety issue.
These days we are all encouraged to be kind, and banter ain't banter unless all parties are amused.
I know you are being advised that your height is regarded as average, but I can see why it's bamboozled you.
Whether this colleague is making these regular comments as a (misguided) friendly banter, OR he's being downright horrible, coz he can see that it bothers you, in fact he's carried on even after you've asked him to stop, but he pig headedly carries on coz he's a nasty pice of work....
.... none of us could possibly know.
Op you've told him on numerous occasions to stop.
Next time, tell him that it's the last time, and if he mentions it even one more time, you'll be reporting him, in order to get it stopped... THAT won't look good on his performance review.
Does he say it in earshot of other colleagues who would be willing to back you up?
Or would he be able to claim that it's your word against his?
Good luck, no one deserves to have any unwanted behaviours, no matter what.
Please take this further, as no one should tell you "how" you should feel.