YANBU. The carers don't mind. Your mum would be disorientated if you brought her to yours, and you say, have to use a comode behind a screen. Your mum would get flustered by the noise of children. Seems like your brothers don't understand dementia, or at least don't understand the stage your mum is at. Your mum recognising your children but not their's shows you visit her enough for her to recognise them and not your brother's kids.
IMO your mum would be happier in her own home and own routine. You could visit her early in the day, but you don't have to, as your mum won't even remember it is christmas day. you need, IMO, to let your children make a noise, your DH to be joining in all the time, not having to drive off to fetch your mum while you try to make the dinner yourself.
And you need to be able to have a sherry as early as you feel like it (IMO).
And if you aren't driving your mum home, you and/or DH can (or older kids can be encouraged to) get the pot washing out of the way before tea time, when more washing up will pile itself into a great mountain. (not trying to tell you how to organise your day, you can leave the pot mountain if you like) One person driving your mum home leaves less, i mean fewer, hands at home. (I am beginning to ramble and should go to bed).
So all in all IMO, it is in your mum's best interest to stick to her normal routine in her own home. You could explain that to your brothers. But you don't have to justify or explain to them if you don't feel like it, especially if you already have said something. Maybe if they don't see her as much as you, they don't really know what's best, just have a traditional view of don't leave mum at christmas but they won't be actually visiting or fetching mum themselves. (And if they did your mum would not know it is because it is Christmas day.)
When I am old, if I get dementia or not, I think I will like my own routine at home, at Christmas, with maybe visitors of my children and maybe Grandchildren if I am lucky enough for my children to have children, but not necessarily, and not out of obligation. With my TV viewing ringed in the specially bought Christmas Radio Times. But I am a bit of a miserable stop at home already.