My formerly estranged brother has transitioned and wants back in mine and my mother’s lives.

Oh for goodness sake! I’d be furious about that, so manipulative.

So he’s manipulated other members of your family to make mum feel pressured to extend the olive branch. Or has he tested the reactions he received from other members of family to give him the confidence to meet mum?

I'd be very suspicious of the timing of all this, waiting until after your father's death (a vulnerable time for you and your mother) to make his announcement to the wider family, only then saying he wants to be accepted back into your lives.

His 'transition' doesn't erase whatever happened to cause the estrangement, nor does it obligate you to forgive and forget if you don't feel like it. I probably wouldn't be eager to allow him back into my life, personally. Your mother can do what she likes, but I wouldn't be guilted into doing something I didn't want to do. The rest of the family can think what they like.

“He wants involvement”

Oh does he indeed? Well, you don’t have to let him have it. You can be supportive to your DM and civil to strange brother when family occasions mean you can’t avoid him, but otherwise, I’d carry on just as you are now. I see absolutely no point in trying to play happy families with a horrible person who has been estranged for some time, and I’d want to keep my DC well away.

A dick in a dress is still a dick.

I would just say, to me you are my brother and always will be, I am sorry if you find it offensive but I can't address you any other way. If you want to be in my life you are welcome but it will be as my brother.

Then if he doesn't accept it is his choice. And got have done nothing except assert your personal boundaries.

Stay estranged and save yourself a whole lot of hassle. He's no doubt after a compliant audience to play long in his theatre of life. Wide berth. VERY wide berth.

Well he's a shit, isn't he, getting his compliant army in first. I can see why she feels under pressure. And he is her child, she may see this as her last chance. But you don't have to play along if you don't want to, just say 'that's nice ' and move the conversation along if she tries to talk to you about him.

It shouldn’t be about what he wants!
So what if the callous Brother has now decided they want a relationship. Put your needs first OP.

You don't need to have a relationship with him or involve him in your family's lives. You don't need to talk to your childern about him.

Blood isn't thicker than water, you don't have to have a relationship with him just because you share the same parents.
If you want to support your mother, that's great but be sure the support she's asking for suits you. It's her choice if she lets him renew their relationship but she can't make your choices for you.
I'm sure you can find a way to support her without having anything to do with him, which if it were me I wouldn't, I'd be telling to get lost, but that's me, not you.

That is your prerogative as their mum and it’s not unreasonable to not want a relationship with him again because of how he has treated you in the past.

Yes, to be fair his late onset transness is a red herring here. The estrangement is because he is clearly horrible. And I can't imagine he has become any nicer, just more entitled. He can want what he wants, you don't have to cede to it if you don't want to. I can see why your mother might want to, and that's her choice, but she doesn't get to make it for you too.

I suspect what he wants is validation, not involvement. All The Girls in his family supporting his notion that he is one of them.

He wants support animals, not an equal relationship with real people.

Just protect yourself. Your mums relationship with your brother is their own, doesn't have to be yours.
Estrangement normally has a valid reason behind it and I presume that wasn't taken lightly at the time, so no, I'd not be in a hurry to introduce him to your children or back into your lives. His wants do not trump your own.

Then have a think about how you want to relate to him. For example
Will you not attend a family gathering if he is present?
Will you attend but be aloof but polite?
Or will you be friendly but not accept any contact over meeting at a family gathering?

Figure out your lines in the sand now before you're caught like a rabbit in the headlights when something is suggested or sprung on you

Base your thoughts on the person he is/was. His gender related beliefs aren't relevant other than the confusion he presents to your children. But he won't be the only trans identifying person they will come across and you have most the influence on how they interpret what they see in the world when they are young enough to still think you know everything

If he wants to re- build a relationship with you, maybe leave your children out of it for now and see how it goes.

@Rimding if he gets in touch and you’re ambivalent/keep your boundaries and he asks why, just keep it to the truth - he wasn’t nice to others/nice to be around and not a good role model for your kids. The trans bit is kind of irrelevant here and his prior behaviour is enough to make you cautious. I’d say nothing for now but watch and observe.

Tell your mum you don't want to talk about him, and change the subject.

Don't reply to any contact directly from him - blank him or even block him.

If your mum sets up a situation whereby you walk into the room and he is there, you have the choice of walking straight back out again, or staying in the room but essentially blanking him or being formally polite and distant (may depend on how many people are in the room).

Ultimately, your mother has to accept your right to see who you choose, and cannot force you to have a relationship with him. If it comes to it, you may have to tell her that if she keeps pushing the issue you will see a lot less of her.

Hopefully friendly just means civil if you are in the same room. He can't expect an actual friendly relationship given his previous behaviour.

Your father’s death may have hit him hard and he genuinely want to get close with you all. He may of struggled most of his life with feeling he was born the wrong gender and want you to get to know who he really is and make amends.
You don’t sound like you would accept him as he is though and that will create a lot of angst for your mum. You've also made it clear he has been awful in the past and that you can’t forgive him.
I would stay away @Rimding