It's not necessarily his height that's making you question yourself, it may be that in your post you felt he was over keen.
He may have been on first dates before, and felt he didn't get a further date because of his height... therefore because he had a good rapport with you, and you came across to him as interesting, he wants you to know that he'd like a second date... he well might've seemed too interested, but because of previous rejection, he's wanting you to know that if you are up for a second date, so is he.
I believe the whole "man taller than woman" thing has been drilled into us from an early age, so ear, that we don't even realise.
Also media would have you believe the same thing.
Similarly, you feel the "protective factor" in a much taller man.
Personally, I have known men an inch taller than I, they have been great personalities etc, but no matter how much I berated myself, I couldn't go to the next stage with them.
Weirdly tho my man that I've been with for well over twenty years is only a couple of inches taller, and I think he's sexy, funny fun, caring and compatible to the degree I never want him not to share my life.
This is why I said it's not necessarily the height alone.
Maybe try a couple more dates, be mindful not to give him false hope.
Lots of inches taller than you, is no guarantee to a happy ending.
Good luck!