Definitely have a read and I hope you find it helpful. It won't tell you what to do but the profiles of the various types of tricky inlaws could be enlightening.
I know it's difficult but try not to criticise his relatives if you can. He's been living with their family dynamic all his life and will be defensive about their actions, perhaps especially if he finds them annoying or hurtful. No one wants to face the fact that their family members can be rude, embarrassing, hurtful, offensive etc. I'd find it horribly difficult and I know their faults all too well but I also adore them and are proud of them and happy to have them in my life. It would be much harder if they were crap people (evil grandma is no longer with us!) and someone I loved and trusted like my husband was pointing their failings out to me.
Having said that, they have no right at all to be horrible or rude to you. You know that and he needs to understand that even if they don't. There's no law that means you have to spend time with them.
If this is something you decide needs to happen then your DH has to respect it.
But it depends on what the issues are, how often you see them at the moment, where this takes place (e.g. Do they behave better when they visit you than when you go there), are some of them better than the others. Another factor is your DPs relationship with them. Have they had a good relationship historically? Is he suggesting you all meet up, you, or them? Has it changed since you had your baby? If so, is there a way back to how it was before? Have you tried to address any of the issues with them or are you stewing over their behaviour and maybe taking it out on your DH which is putting pressure on your marriage that you could both do without?
You don't need to answer but there are so many factors in play.
Difficult bastard inlaws are the worst. It can sap the life out of you and put a HUGE strain on your marriage. But there are ways to cut down on the negative.
I wish you luck.