A tough decision and, of course, it's really up to how your decision will leave you feeling. To some people, it won't be a tough decision: you just 'do you' and don't worry what others think. You're having a great time away so why spoil the good times you're having 'and even DS doesn't want to attend'.
For me, I would go to the party because your son RSVPd that he'll attend which means his friend's parent/s will have paid for your son's share assuming a venue has been booked and your son's been added to their food bill costs, the party bag costs, etc. Again, to some hosts, these costs might be relatively small or insignificant while to others, they might be significant.
I'd also lean towards having DS attend because although you're having a good time, such great moments in life are aplenty, but your DS's friend will only ever celebrate his 6th (or 7th birthday?) once in his life. So to teach your DS about being a good friend and avoid lessons on flaking, albeit at such a young age, is still a good thing.
There's also DS's friendship with his 'close friend' the birthday boy. You would potentially annoy his mum who'd want the best for her DS's party. As some might point out, how would you feel if the shoe were on the other foot?
Personally, I would only stay where you are if you know for sure that the host and her DS wouldn't mind, i.e. that they're an extremely laid back family and that enough of Birthday Boy's other close friends were attending to soothe your DS's absence.