OP legally you need to have a good reason to stop them going. A few newspaper articles about a couple of terrible parental kidnapping cases - when thousands of separated parents bring their kids on holidays to meet extended families every year without issue, is not a good reason. The opinions of some random people on the internet won't count. He can take you to court to force this if you don't have a good reason not to give permission.
But morally, you say his parents are elderly. So this may be your kid's only opportunity to visit with the whole family and create memories and photos and all of that. Surely you need a good reason to deny your child that? Again, many, many, many kids visit eg Granny in Scotland with Mum while their separated Dad works in London, and no one writes articles about it because it's normal and the kids return from their holidays happy to have seen Granny and Scotland. Most parents want their kids to know their family, yet most parents are not a kidnapping risk.
Do you have a reason to think he is a kidnapping risk? Because so far you haven't given one other than "he is only in the UK because he loves his kids, so if he could become a criminal and upend their lives by kidnapping them, he might do that".
If he is a good Dad who loves his kids enough to have stayed in the UK for them, is it possible he just wants to bring them on holiday to meet his family and understand that bit of their heritage?
Of course the kidnapping cases are horrible and unsettling, but they're also a tiny minority of the cases involving separated parents where one parent is from another country (which includes Scotland or Northern Ireland if you're in England or Wales). It's just that they're the only ones written about, because stories about banal but successful (or even banal and miserable) family holidays don't sell papers.