I think you’re getting a lot of stick here OP and hindsight truly is wonderful, a lot of the posters seem to think you should have it, maybe you have powers we do not? 
I think you DF is being unreasonable.
He refused to come out with you all, he also refused to take his cards on the Thursday when he knew that he wouldn’t see you on the Sunday, through his own choice.
When you called to say there was D&V in the house he didn’t ask how you were or ask you if you needed anything. Having one BF baby along with your DD and your DH being ill must have been beyond exhausting.
He’s then lied by omission when he’s spoken to your DU and made out that he’s been more or less abandoned by you when it was his choice not to see you on Father’s Day, his choice not to take the cards and you really can’t help your immediate family getting sick.
If I’m honest I can see why he’s divorced and has no friends.
He is also critical of your choices and how you run your life and he also seems to want the relationship with you and your DC entirely on his terms.
I’d think about leaving him for a while. Maybe post the card or pop it through the letterbox.
If he does ask why you’ve been quiet tell him that you’re upset that he didn’t want to see you on Father’s Day and celebrate with you, DH and his DGC and you’re also upset that he’s complained to your DU but not told him the whole story.
I also think that you need to consider how you’re going to handle the relationship moving forward.
It seems like you’re doing all of the chasing and accommodating and he’s doing very little.
I’d also stop telling him much about you or the DC. I’m in a similar position to you OP. Am diagnosed as ND, DM isn’t but has a lot of traits and it’s pretty obvious where I inherited my ND from
DM is also super critical so I have to grey rock her. She has extremely fixed views on what she considers the “right thing to do” and any deviation garners criticism.
You don’t have to break your back trying to please him. It’s fine to suggest things but if not interested that’s up to him.
I’m probably more your DF’s age and I’d also like to warn you about him trying to make his loneliness your problem. It absolutely isn’t.
Presumably he has the internet, like most of the population. He can find out what’s going on locally that he could go to if he wanted to.
His choice not to socialise and talk to people does not mean that you have to provide all of his social contact for him.
You’re allowed to look after your own family and him sulking when you’ve said that your DD and DH were sick and not offering any kind of help is very telling of how he thinks that he should be your priority.