The problem with this though is, as OP has explained, brother will turn up with the cheapest asda frozen trifle and one cider as his "contribution" and then eat all the nice stuff op has bought.
telling them "there won't be much alcohol" - what happens if she or her dh want a drink - if they open a nice bottle of wine or pour themselves a rum how does she stop brother and sil from pouring themselves a glass or taking a slice of the expensive cake she bought for her family? Smack their hands away? That's just going to cause an atmosphere. How does she explain to a 5 year old nephew that his cousin gets a yummy chocolate pudding but he isn't allowed any, or that they get meat with their dinner but he doesn't because his dad didn't bring any?
Either she also has to limit herself and her whole family have a rubbish Christmas meal with the cheapest possible food and one drink each all day (which given shes said she enjoys treating her family and having nice stuff - which is fair enough at christmas! will just be a bit shit) or she will be constantly on the look out that they aren't eating her "nice" stuff - either way it will lead to resentment.
You can't just "tell" people like this that they need to contribute and expect them to do so fairly and appropriately - if they had that sort of self awareness and decency they wouldn't take the piss in the first place.
And even if they bought some food OP will still be the one running round doing all the cooking and cleaning.
If she tries a "compromise" then the onus will be all on her to constantly enforce the new boundaries, again and again, all day, while everyone else is trying to override them because "it's Christmas, are you really going to deny your own brother a beer?" "Who cares who brought what, we're all family arent we?" Etc.
Her mum, brother and his family have an interest in maintaining the status quo - they aren't going to suddenly see the error of their ways and become decent people, theyll pay lip service to it to get her off their backs and then just act exactly the same as always.