I don't think you are being unreasonable.
Not at such short notice.
It's very rude of them to wait until this point in time to make an 'invitation' that sounds more like a blackmail demand when most people have had their plans made for several weeks, if not months.
Why have they reacted like this? Has there been trouble in the past because you spend every Christmas with your family, or because your PILs are not invited to your house?
My DH's parents disowned him last year. There has been a family argument of sorts between me and them, which has escalated from something very upsetting but fixable into me being diagnosed with panic attacks and depression due to their controlling and manipulative behaviour. But up until last Christmas I was still prepared to make the effort to get along with them and went for a Christmas meal at their house a few days before Christmas Day.
Until in the week between Christmas and New Year, when we made the mistake of accepting an invite to tea at SIL's house and PIL's took offence that we went to see someone other than them.
They told DH that they were finished with both of us and that he was a tiny part of their lives that was over now, that we thought we were better than them and were taking them for mugs and using them. They tried to tell him that he didn't really know me or what I was really like (we've been married for over ten years, so I think he probably does) and they made a real effort to split us up that continued right into the New Year and is still going on now.
Yet they can't understand why we don't see much of them now. DH has been a handful of times, DS and I haven't been even once. I get too upset and stressed at the thought of seeing them.
There are a lot of problems between them and everyone in the family, going back years, but I'm the first to stand up to them and really stick to it. I refuse to let them treat me badly anymore, but my DH has been like yours in wanting to back down rather than lose his family. We have backed down quite a lot during our marriage to keep the peace but eventually enough was enough and to be honest it sometimes felt less like we were guests in their house and more like we were hostages to their dramatics, forced against our will for the sake of peace and quiet. Normal families don't make you feel like that.
It's taken a lot of bad behaviour on their part to make him see that it's not acceptable for them to do this.
I can see why you have backed down but I don't think it's right or fair, especially if you do have allergies and are TTC as well.
I can't see that it's going to be a happy Christmas day for any of you if you are only spending it with them because of threats and manipulation and you are ill and worried about the medication while TTC.
I hope it all goes well for you, but I think they have behaved very badly towards you both and it's disgraceful that you have had to cancel plans for a large group of people to give in to a silly tantrum and threats of being disowned.