As others have said, listen to your gut.
My father was like this. Focusing on DD, lots of 'sit on grandad's lap', come up the garden with me, etc, etc. Thankfully, nothing ever happened - both DH and myself had a 'feeling' and kept her in sight. He was completely dismissive of DS.
When DD was 11 or 12, and started rebelling (for want of a better word!) his odd behaviour towards her stopped. She has since said that she thought his behaviour was wierd and was pleased when it ended.
But, his sister had a friend with a young girl - 5 or 6 and he transferred this, frankly, odd behaviour to her. He was made up when his new neighbours arrived with their two young girls. As far as I know, at no point did anything untoward happen, just an over focus on the little girls. A particular type, mind you, blonde with blue eyes.
His neighbours, however, noticed something 'off' and prevented him from seeing the girls. This pissed him off.
After his death, my step-sister let slip that he had attempted to rape her when she was 15. She believes something also happened with her older sister but has never been able to confirm this.
So, apologies for the ramble, but absolutely listen to your gut!