I don't understand what MNers mean by 'rude' any more. OP's story:
Rude = vulgar & gross: no, not that.
Rude = selfish and aggressive: nope, not that.
Rude = meant to make someone feel bad because....
*being short is obviously very shameful to be: the comment didn't imply shame about the shortness
*being short is obviously inferior to being tall. but he didn't say that it was worse, just different
I can't find a "Was it rude" test for the comment to pass.
It's popular on MN to say "Always rude to comment on someone's personal appearance": I can't agree with that. if I did, I could never tell someone that I liked their haircut, dress, shoes, pretty jewellery, eye colour... life would be so dull without free exchange of personal compliments or even factual statements.
Police officer: "Who was it that pulled the child out of the river?"
Witness: "Well... I really mustn't say, I mean it was a person who saved the child's life. But if I describe anything about them, that would be so very rude of me, wouldn't it! My proper upbringing won't let me be so rude as to comment on personal appearance, sorry."