I've done it with both of mine.
We've just got back from a UK holiday with my 13 month old and 8 year old, we stayed in a Premier Inn in Cornwall, I had a wonderful time. The weather wasn't brilliant but we ate out, had fish and chips on the seafront, went on the beach, visited a farm, did a bit of shopping.
It is definitely harder work with a baby, luckily he's sleeping well now, we ended up staying up late and he actually slept in longer than he ever has before. Dh does his fair share and I carried the baby in the sling on the beach and stuff.
We did an overnight theme park when he was 10 months too and he slept in the bed with us and we just watched TV quietly.
I also did it wth ds1 when he was about 16 months, again we had a wonderful time, we found things he enjoyed like beaches, little rides and bouncy castles.
It's absolutely not relaxing like it is pre children but personally I just enjoy a change of scenery and not having to cook for a few days. There's always things to do even for babies and my dc have always kind of grown up a bit after holidays in a way.