And why not? It's a 'you' problem.
My grandad was my favourite person in the world. He spoilt me rotten, spent so much time with me explaining things, teaching me to ride a bike, to read, reading me stories himself, taking me for long walks, train-watching, talking to me. He didn't treat me as a 'baby', really talked to me, and always had time for me, never chased me away. And he bought me everything I wanted, but that's not the point. God, I miss him.
My own dad is a bit more reserved and doesn't have such connection to my DD, but they love each other, and both him and my mum also spoil her rotten and are good grandparents.
Not all men/family members are perverted pedos, you know.