Hi, Maeinha,
We're in the same position - I'm taller than average, my husband is very tall and our ds1 is in the 85th centile for height. Our ds2, on the other hand, who was 7.5lbs when born, so not particularly small, is now somewhere between the 9th and 25th centiles (for height and weight - he's always been in proportion), having gone up ever so slightly from the 9th centile in the last year. He is currently 3 years and 3 months and still more petite than all the other children his age I know, which I confess to being a bit of a surprise after having got used to having a rather tall older son! However, no-one has raised this as an issue with us or expressed any concern, as he is in all other respects developing exactly as we would expect, is perfectly in proportion (in terms of limb size as well as height and weight), very active, bright and quite muscular.
If it makes you feel any better, my dh's sister was very petite for her age all the way through primary school and a large part of secondary school, but when everyone else stopped growing, she kept on growing and is now about the same height as me. So I have high hopes that the same will happen to my ds2. If not, he'll just have to carry on being cute and cuddly, instead! Oh, and the only thing the HV said about it at his 2 year check was that he could end up as short as 5'6" or as tall as 6' (which is still shorter than his Dad, but nevertheless taller than average like me), so I guess we'll just have to wait and see. Oh, and of course, I do check his height every 6 months or so, as there is still a bit of the paranoid mum in me who wants to make sure he doesn't get any shorter still for his age!!!