Sorry for the loss of your son 
I don't think you've done anything wrong at all, and I wouldn't blame you for just walking away from your ILs totally.
My baby dn died around 15 years ago. I think about him regularly, all sorts of things remind me, and some of them still bring me to tears. I know my grief is nothing like my siblings, never was, and never will be.
My dm was quite traumatised, she's never had mental health problems and I know she struggled with this more than anything else she has every struggled with. She hid this from my sibling, and it was the right thing to do; my sibling had more than enough to deal with. I don't think my sibling knows how badly dm was effected by it, and I would never tell them either.
If your bil feels that your mil was hiding it from you, and then if, as you suspect, your mil was moaning about you there may be a lot more to it than just your bil being a twat.
I'm certainly not saying your bil is right, but if he was trying to support his dm and she was bitching about you, then he may feel that you used his dm when she didn't have enough to give back. He's then lashing out at you (the wrong person) which isn't fair, but I can see how it might happen.
I'm not saying that he is right, but just trying to give you a potential situation where this has arisen.
In all honesty, if your partner is happy, I think I'd cut totally back on your bil/sil. Send your mil a photo/little update, but don't expect anything back. A bit like you might to an elderly aunt who would like to see it, but wouldn't have the energy to respond.
Don't mention your bil/sil to your mil at all, keep the focus on "what a lovely little dd we have". Give her nothing to gossip over. let her bring up the suggestion to visit if she wants that, then meet at a neutral place.
It is not you or your dp's job to create relationships in the family when it is the others in the wrong. Support him in what he wants to do, but don't expect anything from the others.