Work out what you can reasonably afford to save towards it then plan around that. Don't overstretch yourself.
Presumably you will be working for some of the holidays anyway, what happens to the kids at those times?
Do some research in your local area to find what's on cheap, what attractions you might want to go to and what you can do for free. Make a bit of a plan, so perhaps you'd say:
We'll do one big day trip to somewhere very exciting. Entry will cost x, youngest goes free, we'll take a packed lunch.
These are three free things we can do - for us that would be a country walk, maybe the beach, picnic/games in the park.
Chill out and movies at home day - make some popcorn and turn it into an event.
Get the kids involved with the planning, DD and I do this where we sit down together and work out what we'd like to do. You'll be surprised how low maintenance they are in their wants and it's good to manage their expectations too. Resist the urge to provide something all singing, all dancing that costs the Earth.
Another approach is to join the national trust next summer if you're not already a member - for the two of us it's about £10/m. We have a fair few NT properties near us, that's our plan for the summer hols aside from our holiday abroad. Costs wise I'm planning extra petrol and spends for tea & cake.
There are events we'd really like to go to but we've decided together that we'll give them a miss and put the money saved towards holiday spending money