It's up to everyone who they choose to spend time with. Xmas day is important to many so it would also be important that it is enjoyable. Why would you spend time with people whose company you did not enjoy?
I would not, it doesn't matter the reason ie tory voting, leave voting, racist, whatever it may be. I would not spend time with people who I did not like or whose world view and opinions I don't like. And I'm not talking basic opinions but opinions that go to the core of their personality like the ops grandparents.
And since anyone can be abusive or violent, just because they are a parent or grandparent, does not excuse them being abusive and just because you share blood does not mean you have to put up and shut up.
The man I do not call father thought that we had to be there for him as he got older because he got older. Well selfish, abusive people get old too. You don't have to respect people just because they are older or family.
You respect people who show you and others respect. The people that you invite into your home on important days or for a meal at any time are the people who you respect, like, love or who you have other good things in common with.
The tory voting stuff is a red herring but it does speak to their core personality with the updates provided by the op. Unless you also think like this or they respect a ban on discussing their opinions and thoughts then you would not would them to be in your company. Anyone who does is a masochist or a martyr and I am neither.