I know it's a bit different, but Ds has spent every other weekend with his dad/grandparents since he was 2. They took him to India (without me) when he was 4. My mum has had him for a week at a time, 3 weeks on the trot when my dad was dying.
So he loves your dad, they have a fab time, he wants to go, it sounds lovely. Do you have safety concerns or something? When my dad was alive he was a bit daft about child safety things, for example, he once took ds on a fairground ride and wasn't watching him as was distracted with getting his own seatbelt off so 2yo ds undid himself and nearly jumped over the 10ft drop side of a high ride, that kind of thing.