I understand OP, it would hurt his feelings and you love him, he’s your brother.
It is also ok to want to have some Christmases with just you, your husband, and your child. And, your brother should be thoughtful enough to realize that.
This is what I would do. You make yourself a less attractive option for Christmas, and your sister a more attractive option.
Ask your sister to invite your brother to Christmas via text with the suggestion they watch a film he would like, for instance. Have her entice him with whatever else would make him feel very comfortable. Tell her you feel he needs time with her, and find out when she sends text. Make it all very natural, because really, he does need to start having meaningful relationships with the rest of his family again.
On the heels of that text, you call your brother and say you hope he doesn’t mind to help with a big project when he arrives on Christmas Day. Make it a very dreadful undertaking. Don’t give him time to reply, say you have to get off phone, toddler needs you.
(Hopefully), your brother will make his excuses, you haven’t hurt his feelings, in fact he may worry that he’s hurting your feelings, he will spend time getting closer to his other sister, and your husband gets a Christmas with his own little family, and you can too—there’s nothing wrong with that!
If that doesn’t work, backup plan is what another poster suggested, tell him you’re going to your in-laws.
I too spent so many Christmases, but with in-laws who didn’t even like me, just because they demanded it. I wish I’d done sooner what was best for me, my husband, and child.