Op has said that DS has his own hobbies and is off on a sporting holiday soon, so no, it doesn't mean that other family members never go on holiday or anywhere. It just means that up until now, OP hasn't been accompanying them on holiday or trips to the theatre or cinema.
My DS has an ASD diagnosis, we would never think of expecting him to endure a hot, crowded, noisy café when we could equally choose another where we know he is comfortable. We make a reasonable adjustment.
When he was younger, he wore ear protectors so that he could cope in noisy chicken/pizza restaurants. He compromised.
We organise our family holidays around what he enjoys (currently: camping, fell walking and heritage railways) because that's what will make our holidays enjoyable for us all. There's nothing like a truly miserable kid to kill the holiday vibe! DS gets to do what he wants. DH enjoys time off work and fell walking. Because I am unable to do the fell walking, I take a book and some paints so I get time to myself to pursue my gentle hobbies that nobody else is interested in. We all enjoy a ride on a steam train with coffee and cake at a pretty heritage station café. We all loved the Laurel & Hardy Museum telling their history with original artifacts and showing their reels back to back in the tiny cinema at Ulverston. DS had never sen a Laurel and Hardy movie. We all laughed uproareously! For a rainy day, I highly recommend. I digress....
Ok, this last accommodation (organising our whole holiday around him) is a bit more extreme but he's 13 years old. He most likely won't want to holiday with us forever and will soon reach an age when he can be trusted to be home alone while we jet off to Barcelona (That's one of my dream destinations).
Also, 'hangry' is real (even with adults)! It just takes a bit of planning to avoid those affected getting to that state.
My point is that we navigate family life with a combination of adjustments, accommodations and compromises so that we are all reasonably comfortable and get a chance to do the things we enjoy.
It sounds like the café incident was just bad planning OP but now that you have a diagnosis, you can better understand some of the reasons behind your symptoms, and work on strategies to help you cope in some of the situations that you would have avoided in the past such as visiting your daughter. Maybe, noise cancelling earbuds for aural overload, something akin to the spoons method described by PP and a little bit of CBT?
Long and involved articles may not be your husband's bag but he sounds like a nice man (taking you both shopping which he hates) and I'm sure he'll be on board with the positive changes that you can make.