What's the fall-out with your own parents (as it's their mum & dad) if you don't invite?
I mean - is the situation that you're hosting the family Christmas this year, and now your DH wishes to uninvite a significant chunk of your family just because of the election?
If so, that is unreasonable. DH will need to put up with it for one day, decide never to host them again in future if need be. You will need to run interference - no discussion of politics, your DH gets a free pass to say what he likes if he needs to, or to go out with the baby for a long walk to her them to nap, or whatever.
Honestly, your DH (and you) have known for a long time what they're like - the recent 'revelation' of them voting Tory cannot have been a surprise. So to change plans now, so close to Christmas, on a principle, is pretty unreasonable.
For sure, let your DH limit contact in the future. But if you are committed to hosting your family Christmas this year, suck it up and do it differently next year. Support your DH in stopping them goading him, or talking about politics, and get some strategies in place for what you will do if it is brought up.