What you are feeling is very, very common.
Not feeling 'in love with' your firstborn is really, really, really common at this stage. You obviously do love him - that's what the close and protective feelings are. But often the shell shock of motherhood and the chaos of hormones and the everything else is just too much.
It's easy for a grandma or a dad to be 'in love' with the baby. They aren't at the coal face every minute of every day. Their world and their body haven't been (almost literally!) turned inside out. Your husband goes off to work and his day is basically the same - he has a lot of 'old normal' to cling to.
I didn't feel madly in love with my first baby. It was something that grew over time. If you search the archives here, I am far from unusual. More people seem to get those feelings with later babies (though not necessarily). With my second and third I did get that 'whomp' in the heart and instant love. I adore them all now!
The thing none of us can tell through a computer screen is whether what you are feeling is just all of that stuff, or whether you might be tending towards post natal depression, or an anxiety disorder of another kind. If you are, there is no shame in that and it won't hurt your baby. But you do need to talk to someone to find out.
If you do feel distant, do you know what I found amazing - babies can't tell how you feel! As long as you are behaving as a loving mother, they don't care! And other loving adults around really helps too. Mother nature is very, very smart - you can fake it until you make it with absolutely no ill effects!!
But do talk to your HV (or midwives, if you are still under them) if you think it might be PND.