"I want to get along with Inlaws for dhs sake but they are very difficult".
That is not going to happen.
They are impossible and would be impossible for anyone at all reasonable to get along with. It is not your fault they are dysfunctional, you did not make them this way. If your DH wants to speak still to his mother that is up to him (he is still in the fear, obligation and guilt state with regards to her) but that does not mean that you and your DD have to be involved in any aspect of their lives. Your job as parent is to protect your child from such malign influences.
You likely too come from a family unit where this type of familial dysfunction is absolutely unknown so that makes it difficult for you to deal with as well.
You cannot however, apply the "normal" rules of familial interaction to such people, infact the rule book goes out the window when it comes to dysfunctional families like your DHs.
Some grandparents as well really should not be allowed any access to their grandchildren.
And no it is not wrong to not want to speak to them ever again.
A good rule of thumb here is that if they are too difficult/toxic/abusive for you, they are far too toxic and difficult for both your vulnerable and defenceless child. Given their behaviour as well towards both you and DH they bring nothing at all positive into your lives and they will not bring anything positive into your child's life either. The best thing you can do is detach and ignore any histrionics on their part.
I would suggest you read "Toxic Inlaws" written by Susan Forward as that may help you. I would also suggest you post too on the "well we took you to Stately Homes" thread on these pages.
Detach, disengage and ignore. Do not rise to any bait. BTW if they do start sending abusive text messages or e-mails report them to the authorities as that action is a criminal offence. Infact I would now block their numbers from your phone and e-mail account.