Your boy is not yet at the developmental stage to even begin to start understanding what this means, let alone feeling hurt by it. He's waaaaay away from that level of cognitive development. I mean this kindly but this isn't a little bit of projection - it's complete projection. I promise you he doesn't care - or understand.
I have 11yr old twins who were 8 wks premature. When my DD was a year old she was just growing into 0-3 months clothing. (When she was born, she was still covered in lanugo, especially on her back. One of my friends sent me a card with a picture of a chimp and wrote her name across it.)
They were teeny tiny. And it took them many years to start catching up.
My DD is now a strapping girl who is strong, muscular and wears clothing very slightly above her size. She is fit, healthy and well.
My DS is smaller and a skinny little thing but presumably when the hormone surge hits he'll start to develop muscle and fill out a bit. He wears clothing about a year or so under his size height-wise. He is also fit, healthy and well.
Your boy may well end up being the same size as others. He may always be a bit on the small side. Time will tell.
Mums always comment on size "ooh isn't your baby big?!" "ooh look how small and cute he is!" A lot of the time their comments are just a reflection on the fact that your baby is a different size than theirs was/is. It's not a criticism. To me, most babies look bloody massive because mine were such tiddlers!! I honestly can't get over how big a regular newborn looks.
It carries on through school when you see class photos etc. For years my two were the smallest ones in there. If he is small, there's no point denying it or trying to hide it like it's shameful. Who cares? As he grows, surround him with body positivity comments about being strong/fast/healthy/well/energetic - whatever suits. Those are the messages he'll absorb and how he'll view himself.