YANBU!! If you don't feel someone if safe to be responsible for your children, they shouldn't be looking after them! Doubly so, if the situation has only come about from your husband being lazy (rather than, say, if your husband thought it was for some reason important for them to have one-on-one time together).
As a secondary point, as the father is your current husband and not an ex, you also have a relationship issue, in that he should respect your boundaries.
Your MIL is also overstepping, and wrong to boot. Your SIL does not "deserve" anything with your children. She has no rights over them, it is up to you and your husband, whether you think your children will benefit from a relationship with her.
My SIL and DH do not get on, and will not be in the same building. Similar, in that she can be a very difficult person (although as it happens, me and her get on okay on the rare occasions we see each other). One day, she lost her mind, screaming at my DH about something irrelevant, and threatened to hit him. All this in front of the toddler, which is why they now don't meet. She believed she had a "right" to see her niece and nephew too. As a family, we facilitate her seeing the children on occasion, because on balance, we think family is important, and she only acts like that with my DH. But we also have the rule that she can't be alone with them, because ultimately, she is unstable. My MIL and FIL host me and her and the children. They would never dream of going against what we've decided is best for the safety of our children. Sounds like your PILs and DH are scared of your SILs moods, and are happy to offer up your children to mollify her, which isn't okay.