I'm scared they will not want to be friends anymore if I make a fuss
Sweetheart, they're already not your friends :(
I've been where you are with a group. It's a very hard thing to tackle because it isn't that they're awful, or mean. It's simply that you're B-list, for whatever reason, and they just don't care very much about you. You aren't part of the main dynamic. It doesn't mean that they don't like you but it only goes so far.
However, it's a really damaging dynamic to be in for you. Especially as, this isn't about you not being interesting or fun enough or whatever. It's simply the way it is, and with another group of friends you would be part of the core, and cared about, and what you wanted would matter. That's the way it should be.
I dropped my group of fun, inspiring, diverse, perfectly lovely folk where I was one of the periphery people, because I didn't like the way the dynamic made me feel. And I know I did the right thing.
I think it would do you the WORLD of good to send a breezy text saying, changed my mind actually - this place doesn't work for me so I'm going to just pop in to say hi and have a glass of wine then I'll be eating elsewhere. They'll be full of shock and ohhh but we're disappointed, and you'll smile to yourself and think I'm sure you are... but not disappointed enough to consider me in the planning, or offer me a lift, or think twice about me in any way. See ya for a wine then me and BF have reservations at DeathToHalloumi, have you been there, it's ace :)