I competely know what you mean OP, I felt the same when DD was tiny, and do agree that you mostly have to adjust your expectations. But it does get better. With DD we have:
Spent a week in aunty's caravan in Llandudno, DD 4.5 mo - mostly me being stressed about baby things as I had PND and she had reflux and it was mostly a sh!t time, but what made it a holiday was that the three of us were together, DH and I talked, it was still better than being at home wtih DH working long hours, and loads of washing etc.
Spent 5 days with my parents in Spain, DD 7 mo - was great as Mum & Dad helped with DD, we ate out at lunch time and in the evenings BBQ'd at the apartment or ordered lush takeaway (e.g. a big paella platter from the local restaurant), spent time on the beach, sightseeing etc. and DD mostly fit in with that.
Spent a week skiing in Austria, DD 13 mo - again with my parents, they babysat in the mornings whilst DH & I skied, then we met for a long late lunch together and did things together in the afternoons. Ate in the apartment evenings.
After that we have basically done 'normal' holidays e.g. some abroad in the sun, some with my parents or other families, some in the UK, etc. and now she is older (well she's only 2.5 now but it's soooo much easier and more fun) we have a great time together. Holidays are different from couples holidays, but they're not less good, they're great fun these days and DD loves them too and is starting to get that holidays means Mummy AND Daddy all day playing and having fun.
So hang on in there, it gets better, the baby stage is hard for some people (it was for me), but it does get better and it does get easier, and holidays WILL feel like holidays again.
Dx