I am 5 ft 5 ish and DH is 5 ft 10, but tbh it wouldn't bother me if a man was the same height or shorter than me (well maybe a max of 2-3 inches...) My dad was small-ish (5 ft 4,) and he never ever had a problem with it. The ladies loved him! 
He was a real charmer and women fell over themselves to chat to him! He looked a bit like Brad Pitt facially (I'm not kidding,) with a full head of wavy hair and a cheeky smile.
My mother was 5 ft 5, and it never bothered her that he was shorter. It only bothered her that he was flirty! Not much though, as he really loved her, and she looked a bit like Liz Taylor, so she was never short of male attention.. 
So I think a man being short is only a problem if you make it a problem.
You are within your rights to be put off though @dusktildawn, just as a man is within his rights to be put off if a woman is overweight. (Not a dig by the way; just saying.)
My (female) friend is 5 ft 9, and she said it would be a dealbreaker if a man was shorter than her, even by an inch. She said she wouldn't care if he was a bit ugly facially, and a bit boring, as long as he is at least 6 feet tall. 
Personally, I would rather have a man who has an attractive face and personality, and is 2 inches shorter. But each to their own.
I also know several men who don't care what a woman's face is like as long as she is skinny. As I said, each to their own.