You're being a bit precious, to be honest. Virgil is unusual enough for people to not be sure they've caught it right the first time.
It's essentially 'foreign' to most people in this country, so they just want you to repeat it to ensure that they've got it right. I bet you'd get the same if he was called Sigrid.
As for people being "rude" for smiling (which means "its obvious they hate it"), well what do you want them to do? That is a polite social response. Personally, I don't like the name Virgil at all, but I wouldn't want to be offensive so I'd probably smile and say that it was unusual. It would be madness for me to throw myself into paroxysms of delight that I didn't feel just because you're all chippy about his name.
Do you honestly expect everyone to wet themselves with enthusiasm at your choice? You're being a bit loony - you don't get parents of Rosies and Jacks on here griping that random strangers haven't had an orgasm over what they've named their offspring.
Here's a test for you: You're talking to someone in the playground and she introduces you to her daughter called, God - I don't know - Chardonnay-Mae.
You (a)aren't quite sure you've heard it right because it's unfamiliar to you - did she say Charmaine? Charley-Rae? and (b)you're thinking "that poor girl!" because you dislike the name.
So you ask the mother to repeat it. She does. Yep, definitely Chardonnay-Mae.
Do you gush "oh, how gorgeous! What a pretty and original name!" or do you give her "a sort of fake smile as if they dont know what to say."?
You like his name, his father likes his name, he (hopefully) likes his name. That's all that matters. Unless someone is actually offensive about it, don't waste your time worrying about whether other people like it or not.