This is a really tough one, but for me the bottom line is that ultimately if you are not true to yourself and what you believe to be right, things will continue to feel wrong and bad - even if on the surface it's all 'ok' (i.e. you not being cut off from them).
You may not have handled what happened in a particularly good way, but were you in the wrong - I don't think so. You were standing up for what you considered mistreatment of your own sister and nephew. What basically seems to have happened is that your bil 'punished' you for this (which he had the means to do, as he controls your sister - he sounds horrible, by the way!) So ultimately you knuckled under so that you could get to see your sis and nephew. And you are continuing to knuckle under, and that's what's rankling. It would with me too. He really thinks he's the strutting old cock of the walk, doesn't he? He can slag you off (to the children - how horrible, what shitty parenting!) and almost to your face, and enjoy the fact that you won't tell him where to go, because you fear losing contact.
Well I'm afraid that I'd tell him where to go. Which is a bit of a calculated gamble but I'm afraid my pride would force me to take it. Nobody would get to call me a bitch and then expect me to host a party for them. I'm afraid I'd be pulling out and saying that I'd had ENOUGH. That clearly BIL has a continuing problem with you, so he needs to put his money where his mouth is and just keep away. So no free party. And yes, source new babysitters.
Then it's up to your sister what she does. And as it sounds as if she's got more guts than she did a few years ago (going out etc.) then I reckon you refusing to be treated like shit any longer might make her take a bit more of a stand. Also, your DNs are growing up more... and they need baby sitters too, you know... I don't know, but I'm going to stick my neck out and say that if you stand up to this bully, you may have a pleasant surprise and find that he winds his neck in a bit. But even if he doesn't, and there's another bust-up... well, it sounds as if it's NEEDED.
You can't carry on being bullied forever. It's making you unhappy and poisoning the whole set of relationships anyway.