Keeping these people at all in any aspect of your lives is an absolute recipe for disaster as you have already seen (and you've had years of this already). These people are the root cause of your DHs anxiety and depression and they will continue to wreak havoc in your lives given any opportunity.
It is NOT your fault they are like this, their own families of origin did that lot of damage to them.
It is NOT possible to have any sort of a relationship with a narcissist and his parents are not good grandparents. Narcissists in particular make for being deplorably bad grandparent figures, they were terrible as parents to your DH and they will not make good grandparents to your child. Never leave them alone with your child either!.
The actual mechanics of how the NPD grandparent will misuse their relationship to their grandchildren will vary. Generally, they will either over-value or under-value the grandchild as a means to get to you. Often, when they over-value, it is the objective of the narcissistic grandparent to steal the child from you. I mean that in both senses, physically and emotionally. Ngrandparents are known for so much trash-talking against you behind your back to your own child or children that they want to go live with grandma or grandpa, or the Ngrandparents simply inspire rebellion of the child against you. They steal the hearts of the grandchildren. Sometimes, they will battle for physical custody of a grandchild after their slander campaign against you has won them powerful allies. Many times the Ngrandparent has a lot of extra cash to throw around since they are done raising a family. They may successfully exploit the natural selfishness of the child by using cash or toys to lure them.
They do not love your child, they see her purely as narcissistic supply and they could well go onto manipulate and try and control her as they have with your DH. They will use her to get back at you as her parents. Your DH is very much in FOG (fear, obligation, guilt) and is also very much an adult child of a narcissistic parent. These people have done an awful lot of damage to their son.
You ultimately need to move and go no contact with these people because they will never give you any peace nor respect any boundaries you care to set them. They are already turning up without prior arrangement and are extremely demanding.
One thing you can continue to do is to keep on refusing to budge to any demands to see any of you. If they cannot or will not behave decently which they do not anyway they get to see none of you. You will need to support him, I do not think he is at all strong enough to withstand their continued onslaught of his family unit. You need to protect yourselves from such malign influences.
You would not have tolerated any of this from a friend, family are no different.
I would suggest you read "Toxic Inlaws" written by Susan Forward and your DH should read "If you had controlling parents" written by Dr Dan Neuharth.