I felt that way with my first and the name I chose for my son I really liked, but not as much as the girls name I had in mind. He's 6 now and I love his name because it's him. It suits him and I can't imagine him being anything else.
Hope that helps...
FYI - I'm having the same problem this time round! However, the other way around! I love our boys name, but I'm not a lover of any of the girls names we've considered and my gut tells me this is a girl! Funny as the name I LOVED for a girl last time, didn't even make it on the list this time!
Mine was a family name too! I think what's changed it for me is the fact that I now know more children than I did before my son and I know a couple with the name and well, I'll keep it polite, but it's put me off! ;)