Why are people so rude?! My little boy is Alfie and not that it matters but he is not from the background that has been described in the slightest. Quite the opposite in fact! It just happened to be the name of my much loved grandad and I always wanted my own little Alfie. In a few years to come there will be hundreds of Alfies everywhere from all different backgrounds - it's no different to Charlie / Freddie and I happen to know a few of them that are studying masters degrees!
OP - both names are lovely, don't listen to shallow minded people on the Internet, just go with your heart.
Ps - obviously Alfie gets my vote!