I find this intense as well. I'm also very a small family - just me and my mum, and I don't even have her anymore, so it's just me. Dh has a big family. I don't mind them in small doses - I can do 24 hours. Anything longer than that, I find really difficult. They are perfectly nice, but they are the sort where all interaction is 'small talk'. I can only chatter for so many days about safe topics like the weather, the garden, what year some cousin's child is in school, how someone's planning application for their shed is going, etc.
The clincher with us is that they all come to us because we are the only ones who could host everyone, so no one would see each other if they didn't come and stay at our house to do it. I've started to just say, family can visit, but I'm not changing my plans. I busy myself with things I want to do and let dh get on with entertaining them. And I limit the days they are welcome to stay.
In your case, I think it's perfectly fine to say to your partner that you don't want to attend every gathering, or if you do, that you'll travel separately. He can go for 3 days or whatever, and you drive up for just the day and an overnight. Or you put your foot down and say you're staying in a hotel. They might think it's odd, but they'll have to get over it. You could always use the excuse that you've been not feeling well the first time so you wanted somewhere to rest where you wouldn't bother them.
I do think, to an extent, you need to suck it up and take one for the team if you want a happy relationship. Sometimes you need to grin and bear it for someone else to support them in something that matters to them. But you can make it more bearable for you with just a few quick fixes.