Gosh, your baby's so new he's hardly had time to think, let alone hold deep and meaningful stares with you.
He couldn't see properly for the first 6 weeks and, since that time, his sight has been growing stronger.
I feel his 'lack of interest' in you may be because he's got far more interesting things to look at rather than someone who has their face in his most of the time (from changing/feeding etc :)
There's a whole world out there he hasn't seen and it must be far more interesting than looking at you (sorry!).
He will, in time, be more interested in you. Indeed he will display excitement and happiness when he sees you, but for now he's probably just exploring.
Health Visitors, especially inexperienced ones (and most of them have never had a baby so hardly know one end from the other :o) can sometimes have very little tact when they open their mouths before they think. As if she should have said that!! It's something for a doctor to diagnose, not a fishwife! (or a HV!!).
Go back to the dr, if he doesn't have a definitive answer for you, then ask for a second opinion.
And try not to worry :)