Sorry to hear your brother is having a hard time - his situation is not uncommon. It's always difficult to break free, so good luck to him.
The usual way of achieving what you describe is in a parenting agreement, which both parents develop together. There are good templates online, that allow parents to set out how they want to work together to co-parent. The kind of things that are often covered include not speaking disparagingly about the other parent; not blocking access to telephone contact etc during their time with the kids; discussing major decisons about schooling; education etc before they're taken; how handovers will work; who will do what by way of medical care / school appointments etc - essentially, anything that either parent thinks could be important.
Be warned, however, that this really depends on goodwill and maturity from both sides. My ex suffers from mental health issues that mean she can be pretty emotionally unstable. On a good day, she'd have no problem living within the framework of a parenting agreement. On a bad day...not a chance. So, it's worth a shot, but don't pin too much faith in it.
The best thing your brother can do is ensure he secures 50/50 residence, so that the kids always have a calm, stable, predictable, and loving environment for at least half their time. And be prepared to act, through the courts if necessary, if the mother engages in behaviour that is harmful to the kids.
In my case, I had to involve the school in safeguarding the kids' welfare in the early days, and through them Social Services (they were aware of us anyway, because the Police made a referral to them after she assaulted me). When she realised I wasn't messing about, and was prepared to take official action to safeguard the kids' wellbeing, she moderated her behaviour to some degree. So he needs to understand all the support and help that is available to the family, and access it when needed if the parenting agreement isn't enough to ensure that the mum acts appropriately towards the kids.