This is a masterclass in how NOT to be assertive. You missed so many opportunities to just. Say. No.
I got the message about hosting through DH
"No DH, we are not hosting this party, you'll have to tell your parents to make other arrangements."
my IL finally messaged me directly and asked if I was aware of the plans, I said it was mentioned.
"DH told me about the party idea, I'm afraid it's a no from me, I'm sure you understand."
I asked about all of that and if invites had been sent out - the answer I got was basically yes invites were out (I am yet to see it)
At this point would you not go apeshit?! They have sent out invitations to a party at your house without your knowledge or permission! Where on earth is your backbone?! At this point you get angry and tell them to contact everyone to cancel/change venue or give you the guest list so you can do it yourself! And if they refuse you insist on DH moving out or you move out yourself!
I said I'm not going to be around for it because I'm not comfortable with it
The response was "ok in understand". Not ok we'll move it or cancel.
What on earth makes you think that they were going to decide to move/cancel based on their behaviour up to that point?! You don't say you're not going to be around, you say that the party is not going ahead and you won't be letting anyone in!
as newlyweds I'm trying to respect his family even though it's clearly not being reciprocated.
why why why? Stop it. Stand up for yourself.
Hopefully there won't be a next time because my attitude will be complely different should such a request be mentioned. And hopefully me staying upstairs will be a clear enough message!
Of course there will be a next time, this time they've done it and got away with it, and the message is that you and DH allow it to happen and you stay out of their way, allowing them to take over your house.