Thanks all, I forgot to update these past few weeks. Yes @LindorDoubleChoc it is FOG, fear obligation and guilty (as opposed to fats, oils and grease which came up when I googled it!).
I am very annoyed with DH, because I am a terrible people pleaser and cannot cope with people not liking me, its so silly and childish but its the way I am. He should know that a throwaway comment about someone maybe bitching about me is enough to put me into a tailspin. I am still annoyed with him, but the truth is I already had that worry. Dad's girlfriend has never been to my house for over a decade. I have invited her several times through him and he always declines, saying she has plans or whatever and I'm pretty sure he doesn't pass the messages on. She has only met my kids a few times yet she often buys little appropriate gifts for them and leaves them with my Dad. She never had kids of her own and I think there is a part of her that would love to be a stepgrandmother but he isn't letting her in. I've often been tempted to invite her to something directly but I think that would be really inappropriate, I'd be furious if roles were reversed.
I caught up with my aunt and said how I was feeling etc and funnily enough she has had the same experience with him. She said she phoned for a chat the other day and he made up some lame excuse to get rid of her, she said she knows what he is like! Its funny the judgment there seems to have been entirely in my head. I'm glad we had that frank discussion.
@PermanentTemporary yes its a MN cliche but I think you struck the nail on the head about neurodiversity. I had a lightbulb moment a few years ago relating to this. Because he is in his 80s, a lot of behaviours are typical of an elderly man living alone. However my memory of my Dad as a younger man is what we would call ND these days, I'd say high functioning autism. People describe him as quirky, eccentric, odd etc. Looking back I think he found living with a wife and kids really difficult, his way of coping was to bow out of family life. Not that thats ok, it just explains it.
He also is subtle as a brick and very much prefers my sibling. He isn't critical of me or anything, just prefers them. He did the same with my kids, he never liked one of them and fawns all over the other, even buying a present for one and not the other. Kids aren't stupid, my eldest said from a young age that he knew Grandad didn't really like him much. This is something I found quite unforgiveable and still struggle with.
I've been to see him a few times recently and tried to be helpful in practical ways, offering a lift somewhere or bringing down a few dinners. I think this works better to touch base rather than a visit. The last visit I stood in the kitchen in front of the cooker for 40 minutes and could almost sense his relief when it was time to go. I've also decided I won't ever 'pop in' without notice but I might be a bit more pushy on when I will call, and just keep it frequent and brief. And i need to make peace with the fact that he doesn't really like my company!