Your mistake was posting in AIBU, where people love to cherry pick bits of info and ignore others, project, and feel they have a free pass to lay into strangers.
No, YADNBU.
This man has other Christmas Day options. He needn't be home alone on Christmas Day.
You will be doing all the work, so it's not up to DH who gets invited. If he was doing it all he'd have more leeway to make unilateral decisions. If he 'doesn't get Christmas' it's unfair to reap the rewards of you cooking and prepping and organising, but then minimise how big a deal it is by inviting people without asking, or not bothering with helping.
As a couple, the decision making process was inequitable. You consulted, he didn't. Unfair. If there are no consequences, it communicates that you are ok with this.
As hosts, you are providing an experience for your guests. If you care about that kind of thing, that usually includes providing somewhere warm, comfortable, with food and drink you think they'd like. Like not giving meat to vegetarians. Or peas to someone who hates them. Or strangers to shy introverts. I think it's pretty gross the way that some PPs are calling them miserable, boring, losers etc. You PPs are being extremely unpleasant.
It also sounds like you don't think he has a lot to commend him anyway. It's Christmas! Why have someone you don't really like, who isn't going to contribute help or express appreciation in the usual way, in your home on this extremely special and family focused day? I wouldn't. Especially when they have other options and it's going to cripple the atmosphere.
Why did your DH invite him? Was he drunk, passive aggressive, or 'thoughtless' (which really means he selfishly couldn't be arsed to think about the consequences of his actions on others). Is he sorry or concerned about the situation now?