My mother got with my stepfather when I was 18 and it was the most difficult relationship of my life. And that's saying something, as my father and sister were also impossible. Thank god I never had to live with him! He was a grumpy sour selfish shit who made it absolutely clear to anyone who my mother loved that they were not "good enough". Visiting was always really really painful. But I tried and tried, bent over backwards to be nice to him, so that I could keep visiting Mum without him becoming outright nasty. So he actually thought we were close. Meanwhile, I was trying not to notice what a complete arse he was.
My mother died at 67. SF tried for a few years to lock me down, even asking if I could consider him as my father! (I said politely, thank you, but I'm 40, I don't need another father). Then he made a shitty comment about my mother - one of many over the years, also before her death - and all of a sudden I thought, "I have no reason to put up with this HORRIBLE man." So I stopped.
And I am SO glad I did!
Now I'm in my late 50s and I'm realising what my mother went through with him. A second cousin of mine gave me letters that Mum had sent to her mother over the years and reading through the lines it was clear that he had been an absolute misery for my mother. He was really sexually coercive to her, held his money over her, occasionally threatened to throw her out if she didn't do what he wanted, he was bitter, carping and negative about her whole family and all of her friends. He was a coercive abuser, and I am SO glad I have absolutely NOTHING to do with him and my kids know nothing about him.
And this guy, his own kids also rarely visited - which he blamed on his ex - and the last I heard he was trying to use the threat of disinheritance to force them to come round more. Utter rotter.
OP, drop the rope. Become very busy. Your H will not have the social oomph to keep that relationship going, and after a while you can tell your kids things like, actually I didn't really like the way he treated granny, I prefer not to see him too much. They'll figure it out.
You don't need to waste your time or energy on such a person.