So this is not somebody you want in your life, regardless of the trans stuff?
I’d be civil but distant - sort of like you would with an acquaintance that you don’t much like and don’t want to become friends with. Think of him as one of your mum’s faintly annoying friends from church (or bingo, or whatever it is she might do that doesn’t involve you!).
For the sake of your mum it’s good if you can avoid drama or actually cutting him off in a big row, but there’s no need for you to have him in your life or have your kids around him.
If challenged on him wanting more time with you, wanting to see your kids etc, I would just say that given the history between the two of you, you’re taking things slowly, and at the moment this is the amount of contact you’re happy with. Don’t make big pronouncements they can argue with, and leave it open that if he behaves well then in future the relationship may improve.
Never never get dragged into a conversation about the trans stuff, because if you do not completely toe the line he will likely get explosive and cause a huge row.
You can use his chosen name (I think people are allowed to choose their own names however much you don’t like their choice!) and avoid using any pronouns. If he notices you’re not using he/she, just say you don’t want to get it wrong so it’s easier to use his name.
If directly asked for your views on the trans stuff you can simply respond that you know it’s a controversial topic, it’s obviously important to him, and in the interests of family unity and building your relationship you don’t want to get into any big debates.
FWIW I have a sibling I don’t like, and tolerate occasionally in the interests of family harmony. If they came out as trans I’d roll my eyes, but it wouldn’t change anything about our existing relationship.