In fact, I'd say he actively does not want to bother with them for some reason.
He sees them as an inconvenience that take up my time and prevent him from getting his attention 100% of when he wants.
I visit with my daughter aged 5 and he ignores her. In the past, we've had a huge row because he once said he wanted me to do something for him (nothing major!) and then when he saw she was with me he said "what did you bring her for?".
We're older parents - Im 50. When wife was pregnant he said to us - she was there - "why are you having another kid at your age?", "what are you thinking, they cost money". I think this was the start of it.
My wife doesn't speak to him now and I can't blame her.
Surely its not right to be so preoccupied with yourself and what you want and not be interested in your grandchildren?