We have had 10 holidays over the years near St Gilles-Croix-de-Vie so not far away. Unless you want to drive to Ile de Noirmoutier, the best beaches are at St Gilles-Croix-de-Vie and St Jean de Monts.
You are almost next to Le Grand Defi which has high ropes, low ropes, disc golf and playgrounds. We went there with 3 other families with a wide range of children's ages and had a great afternoon out.
Parc des Dunes at Brem Sur Mer is another place worth going for some fun.
Talmont Saint Hilaire has a castle which has activities (pony rides, dressing up, jousting and more) as well as visiting the castle. The activities vary depending on the day of the week so check out their timetable.
The zoo at Les Sables d'Olonne is very good.
We've never done it ourselves but friends have had a great time on the Velo Rail at Commequiers.
Challans has a festival celebrating the olden days in July and August (from memory on a Thursday) and there are craft stalls and performances, with everyone dressed up in old-fashioned clothes, plus places to pick up food.
If you fancy a water park, there is Atlantic Toboggan.
If you want a theme park without rides, there is Puy de Fou although you would probably need to book that in advance. It has lots of shows from through the ages and is a long full day.
The drive from Caen/Ouistreham is very straight forward and there are no tolls. You come off the ferry and head towards Caen, then join the Caen ring road until you reach the A84 junction. Take the A84 to Rennes and after the Rennes ring road, you are onto the E3 to Nantes. From Nantes, you are dependent on a sat-nav as there are multiple ways to go depending on traffic.
The drive through Normandy is beautiful and at one point if it's clear you can see Mont St Michel in the distance.