40s? 40s? Hurrumph! I'm mid 30s actually! Pfft!
It would put me off because I do remember the reference, as others on this thread do. HOWEVER, it would be a daft reason to not use a name you love.
I once knew a John Thomas, so his parents really had no problem with penis associations!