My dad did the exact same thing. spent the first few years saying how she wouldn't let him see us, he'd lie to her on the phone if she called him when we were out with him. Ran her down for years at the beginning. Until he told us they were getting married. He genuinely didn't understand why I was confused. I told him at the time that if he was happy and loved her, somehow I'd find it easier than him running her down then telling us they were engaged.
She banned us from the house in the late 90s. 18m ago, he said he'd work on an invite to the house that year for christmas. Using my 18m old DN as a catalyst (his words) The fact that my then 4yo was also home in the UK for the first time at christmas since the christmas he was born was clearly immaterial.
I called him out on it. all of it. The fact that I was now older than he was when he left us for her, that now I had a DS where as he had 2 teenage DDs just meant that I now knew exactly what he had done, and exactly what kind of person it takes to walk out on your DC and to lie to them and not stand up for them. Course he had a different recollection of history, called him out on that too! 
Nutshell, I told him to stick his invitation. Sis went,and so did the OW youngest daughter (to even up the scores no doubt, but I'd not have been happy with the intrusion had I gone)
I've said my piece to him now, I'll never get invited again, I'll not introduce my DS to his W, why would I? She is not good enough for my DS to meet. I don't want to introduce him to people as low as her. She is an OW, OW don't get to enjoy the original family. she won't get that recognition from me, nor my acceptance. ever. she was complicit in pulling my family to pieces, and further more prevented our dad from spending time with us, lied about us and did all she could to prevent our relationship from thriving. He also did nothing to stand up to her.
He knows I see him because he is my dad, for no other reason that that. He knows now that somehow he can con everyone he likes into thinking his family is respectable, it is not, nor will it ever be. It was founded on lies, and betrayal.
Funny enough I get the impression her DDs are not 100% comfortable with the arrangement either, the youngest one apparetnly only calls when it suits her. Kind of rings true to me. Had my mum done what hers did, go after and shack up with a MM when her own H had left her for an OW, knowing the pain that it had caused, but still going on to do it yourself? I'd have said something at the time to my mum and I'd never have held her in high regard somehow.
Let it go OP, it literally is not worth it. he is weak, he will never stand up to her. Your only hope is that she dies first. Then perhaps you can spend time with him, without him looking at his watch. What a truly sad little man.