If it's a hot weather holiday, I just wear beach dresses and bikinis in the day time (these are fine for wandering around cities, having lunch, museums etc too) so I have 3 or 4 of these that I only really wear on holiday and store in the suitcase anyway. I got the more casual ones from Gap, but also got a slightly smarter one from M&S.
In the evening, I have some dresses that I wear at home and just take a couple of them, and also have things like cropped dark skinny jeans that I wear with a vest and little cardigan, but would also look good with a scarf. That's fine for the sorts of restaurants we go to.
To remind me of what to pack, I have packing lists saved on my laptop, honed over multiple trips, customised for different types, eg summer beachy holiday, city break, work trip, winter sun holiday where I might need warmer clothes for a cloudy day or cooler evening etc.
Also make use of the clothes you travel in. The right trousers/jeans, jacket and shoes can also be worn on holiday, and also save on weight in your luggage.