yes. Height is irrelevant to me
From an evolutionary stand point, many species are sexually dimorphic, having one sex much bigger than the other. This is advantageous, because one large parent gives offspring a measure of physical protection, and one small parent means that parent requires fewer resources, so there is more available for said offspring.
It is not always the male that is bigger, in may species, including most birds of prey, the female is significantly larger. It doesn't matter which sex is bigger though, but having two different sized parents is advantageous.
Hence why it is instinctive to some extent, for human males to seek out smaller mates, and human females to seek out larger ones.
But people vary in how much they are influenced by instincts, and how much they want to be.