I've posted before about a conversation I had years ago with a TW.
They were very hurt by their wife and children's difficulties in coming to terms with their "authentic self" and spoiling their happiness.
Their point was "I'm still the same person after all".
They simply couldn't accept their happiness had come at a great cost to their family and that no, they were not the same person their wife and children thought they knew.
Either they had been lying for years by failing to disclose their gender identity and in doing so selfishly denied their wife the ability to know who she was marrying and having children with, or he was lying now about the reasons for wanting to transition.
His wife and children didn't look at Lola (made up name) and still "see" Len. They saw a total stranger who looked exactly what they were - a man in women's clothes who was demanding to be called by a new name and had renounced the word father.
Imagine what that does to a child. You can't call me Dad anymore....but then be hurt because the child doesn't "see" the "same person they have always been".
The person they thought they knew wouldn't have behaved in this way, been so selfish and narcissistic. They had become a very different person or had spent over a decade pretending to be someone they weren't.
Your brother wants to have his cake and eat it and frankly he's not presenting as NB - he's presenting as a TW.
I'd be supportive of your SIL and tell her to get the hell out of this marriage sooner rather than later.
She might be worried about the children but they will do better with her than if she tries to stay. He may apply for contact but it's hard to enforce if the children don't want to see him (and he is bullying them re: pronouns) and tbh he's likely to be far too immersed in his gender euphoria to give a shit about them anyway.
Personally I'd pack my bags and move as far from him as possible to start again.