My formerly estranged brother has transitioned and wants back in mine and my mother’s lives.

He might think he'd 'like' a relationship with you - he can like what he wants, that doesn't entitle him to one.

This sounds a bit like 'whole new me' aimed to erase all previous history, or an expectation that transition opens previously closed doors and requires indulgence or sympathy. It in fact doesn't change the history at all, or him as a person, or your feelings in any way. If you didn't want contact before, why would him transitioning change that situation for you? You're not obliged to do or feel anything different OP, the history and your judgment to this point remains unchanged.

If it's a case of approaching you with a sincere attempt to mend fences then it doesn't matter what his pronouns currently are or how he's presenting, it's how you would feel about exploring that, and how sincere he turns out to be. If you then find it goes well and you want to mend those fences, that's lovely. But I would unfortunately be wary of whether this was being invited into new games as it can be, or whether refusal to do what he wants now has a new way to deal with it, by calling it 'transphobia'.

Even if it all went well I would also be very wary of bringing children into all this, with all the history involved, until I'd myself done a lot of building the relationship back up, until I was certain it was going well, and that it offered the children long term stability in the relationship, and I felt in a certain enough place to think about re introductions. If he's actually serious about caring about you and the children then he'll understand this, support you and be keen to put the time into building up trust. If he wants to rush to contact with a lot of guilt involved? You've got your answer.

OP, he has been horrible innthe past.
You do not have to move on or have anything but the most surface of interactions if you do meet among wider family.
You do not have to see him at your mothers home.
You do not have to receive him in your home.
You certainly do not have to allow him near your children.

Tell your mother you support and respect her choices, but the staus quo suits you.

Just because he wants to have a relationship with you again, doesn't mean he gets to.

You 100% have agency.
I wouldn't mention his transitioning at all.
Nothing to do with you, you are not in contact.

Canvasing wider family before your mother is highly manipulative.

I would hold strong and be calm and clear with your mother that for you, things remain as is.

If your mother attempts to put pressure on you, she is no better than him.

If you wish for an excuse, you could say you will NEVER get over his treatment of your father, and have no wish to be in contact going forward.
He cannot argue with that.

If you were trundling along ok in the status quo when you were estranged there's no need for you to be back in contact now. If your Mum wants a relationship with him she needs to see it as separate to YOUR relationship with him. You can say to her that he may want to reconnect - but you are quite happy to carry on as you were. And neither of them get to make that decision for you.

I don't have experience with an estranged brother coming out as trans. But I do have experience with relatives who have lost a spouse quickly thereafter being pressurised into a relationship with an estranged child. And it's not gone well. In each case the child has been a real drain on their parent, none of them have given more they've taken. It's been heartbreaking to watch.

I don't know what would work best for your mum, but I would remind her that she's in the throes of grief, and that she doesn't have to move at your brothers pace. If you feel up to it, you could offer to run interference with some of the relatives who your brother has tried to get on side. Reminding them that your mum has just lost her husband and your brother doesn't seem to have changed his spots even if he's changed his dress.

Agree with others that from your perspective, holding the line that you are happy with the estrangement and don't want a relationship with him seems better than addressing his transition and your GC views with him.

I'd expect him to start pressuring your mother to get you on board, so in your shoes I'd be thinking hard about what boundaries you want to set. Then tell your mum what they are; and think hard about how you handle it when he starts bullying her to lean on you to breach them.

I hope your mum is grounded!

Even before this, he was nasty enough for you to want nowt to do with him. Keep him well away from your kids at the very least - he'll send them trans kiddie books just to spite you.

Is he back because he thinks there is money following your Dad's death? If, as you say he is nasty then he may be sniffing around your Mum while she is grieving with the intention of getting what he can out of the situation

This sounds dreadful. Maybe you can be the side your mum can’t express. See him but just treat him as you’ve always known him.
With narcissists in my family I keep organised focus on other people and topics (planning events or visits to coincide with their visit) so they any take too much attention.

Male or female he’s still an arse.

Annoying he’s doing it near Christmas. He sounds very attention-seeking. Chickens always come home to roost and I doubt his relationship will be very healthy either.

If you can maintain a bit of emotional distance (with occasional venting to trusted people!) and stay rooted in your own life and family, it might just be interesting.

I don't share your views, but I still wouldn't be thrilled that an estranged sibling just felt like they could decide they wanted to rekindle and expect me to agree. I think my feelings would be the same regardless of transition
"I am an adult and perfectly able to be civil when we cross paths, but I'm not interested in developing a relationship beyond that."

What was he like previously- does he like drama? Is he wheedling his way back in because he’s bored and likes creating upset?
Is it attention? He likes the attention and wants to come back as this ‘I was a shit because I was unhappy and poor me.’
Could it be inheritance? Now your dad has passed and he wants to make sure he is still in the will.

Nothing you do or say will be enough, unfortunately. You have to accept that anything you say can be twisted so it is presented as incivility.

Grey rock is your best course, honestly.

He was so bad before you became estranged. Its unlikely in the extreme this new circumstance will have improved him or made him easier to tolerate. Staying estranged seems the wisest move.

I don’t know how to approach a discussion with him re his new identity and my personal beliefs and boundaries. I am not going to lie to my children and say he’s their Auntie or use his frankly ridiculous new name. I will not describe him or his female partner as lesbians.

Well as he's your estranged brother who you don't much like, you don't need to pander to him. If it were a colleague or your own child it might be trickier, but this is quite straightforward. No-one can sack you or haul you up in front of some HR panel or a social worker or something for refusing to 'respect' or acknowledge his gender identity.

Continue to call him by his original name and treat him as you always did. Don't invite a discussion about it, but if he forces one then simply say 'I respect your right to dress and present as you like, however I am not going to lie to my children and pretend you are now their auntie. You are not a woman and I will not pretend that you are. You are a straight man with a straight female partner and you are both being disrespectful to genuine lesbians by calling yourself lesbians. These are my personal beliefs and boundaries I will not bend them to suit you. If you want to rekindle a relationship with me in full knowledge of how I feel then let's do it. If you expect me to put my beliefs and deeply held principles aside in order to indulge you and in order to have a relationship with you, then no, I won't be doing that.

It's up to you. Take it or leave it.

Regardless of her actual reason for not wanting to see this estranged sibling, which OP has been absolutely clear was because of callous previous behaviour by that sibling, if she doesn't now fall on the neck of the person she has avoided for years for good and sufficient reason, she will be branded transphobic by that sibling and possibly by other members of her family whom she has heretofore had no reason to wish to avoid and with whom she has never fallen out.

This is a real risk, and will be caused entirely by the sibling if it happens.

The lecture she has been treated to on this thread is a sample of the way such people react to any dissent from their beliefs.

I assume, after your dad had died?

So he picked off all the minor relatives one by one, so he had them all be-kinding your mum, and now he is expecting her to lean on you to be kind. Because of thevpower of social pressure.

It is the trans movement in microcosm - pressure the not-that-interested relatives into acceptance, in order to pressure mum, and then bully the hold-outs (you) with numbers

If you don't get on with her and she doesn't apologise for having upset you, I don't see why you should get in touch.

But shutting someone out of your life because you can't see her as a normal human being who's just being honest about who she is? That's just ignorant.

Your 'beliefs' won't stop her being trans, and your kids deserve far better. You ought to be careful, as they're far more likely to be accepting of normal human diversity and your words could seriously backfire on you.

Just treat her as an individual - someone you either want to see or don't - but don't base it on hating the minority she belongs to. Your children are likely to be shocked, your sister will dismiss you as hateful and your mum will be horribly upset. Why do that? It's utterly pointless.

Sanctimonious tripe.

@roomview well said