I can totally understand your dilemma.
My parents were abusive. My father never got anywhere near my children, however before his death two of my siblings did allow him to know their children.
This isn't as simple as you just saying no. You need to have a proper, calm, conversation with your husband and work out, together, how to handle
Becoming a father will have churned up a lot of emotion for your DH about his own childhood.
You need to talk to him about how he sees his relationship, and therefore your DC's relationship, with his father developing from here. What is he thinking and what are his boundaries?
Then you need to think about yours and then, and only then, can you work out together where you can both settle on a happy medium.
For example of my two siblings, one allowed complete access to my father, weekly visits, no restrictions and he attended birthday parties and christmas dinner.
Whereas the other had a very strict set up - they met ones every two/three months, never in the family home, he was never left alone with the children and he was warned if he ever tried to create time alone with the children then the relationship would be over. Also in that second set up my brother met my father with the kids in a cafe or park, his wife refused to be a part of it. There was no birthday parties, or christmas day invitation for him.
Do not underestimate how difficult a time this is for your husband - his emotions will be all over the place. I ended up having counselling when I had my girls because the whole thing with the past and becoming a Mum swirled up a lot of issues trying to work out my childhood.