I'm a child-minder and have a different perspective on this.
First of all, grandparents have very, very strong bonds with their grandchildren, which parents sometimes dismiss - however, their experience and confidence with a baby can be a bit shaky and if the parents (often the mother) is nervous over this, hovers, constantly asks to have the baby back, and/or hands over the baby with reluctance and anxiety, the baby will be feeling it and become increasingly nervous at being held by someone else - that in this case, a someone else who cares very very much about the baby.
I hear this often when I settle new babies in - a mum (who has no emotional attachment to me) will hand me a baby with reluctance, and will want the baby back at the first sign of being upset, fretting, etc. however, if the mum is confident, trusting and shares comforting words with the baby it's much more likely that the baby will be happy to be held by someone else.
And, it will be much better for you in the long run if you help build trust between your baby and other members of the family/friends.
If it helps, show granddad how to hold the baby, giver him a special blanket to put on his shoulder, and it really is not a big deal if your baby cries for a few minutes. Just imagine if you'd have twins, and one would have to cry occasionally when you'd look after the other twin. Or if your baby was the youngest of 5! They would have to cry occasionally...