Is it family OP? My sister invites us every year but there is no pressure to attend, and she isn't offended when we mostly say thanks for the invitation, but we're having a quiet one this year.
If it's your sister or similar, can't you say truthfully- we find it a bit overwhelming, with all the people, so we'd rather have a quiet day on our own, and maybe catch up with you another day during the holidays