We had a lovely holiday in the Vendee last year. Some of the highlights were:
The market at Challans on Tuesday mornings. It is a large animated market which includes horse and cart rides, big slides for kids, people dressed in traditional/peasant dress, music etc. We didn't buy groceries as such but had a wonderful time wandering around enjoying the atmosphere.
A canoe trip in Sallertaine (la Route du Sel). A very gentle paddle around the canals. With a one yr old you'd probably give the canoeing a miss but the village itself is full of arts and crafts shops which was great to take a lazy walk through and pick up a couple of souvenirs. As evening settled there was live music in a restaurant and the village dance was being set up - locals said we were most welcome.
Taking the Passage du Gois at low tide to Ile de Noirmoutier. Ask about the tide time and it's a great experience for the kids to cross the sea and observe everyone fishing along the way. You could stop yourselves if the timing and weather are right. The island is great although finding a free table at a restaurant was difficult for us (2 families looking to eat towards the end of traditional lunch hours). The beach there was beautiful.
We found that the beach at St-Jean-de-Monts was windy and couldn't buy a wind break from any of the shops a walkable distance away so don't forget yours. Our landlady recommended the beach at Le Golf (less people). You follow the signs to the golf club, park in their car park and walk around the restaurant, over the dunes and the beach is there. You don't have to be member to eat at the restaurant (not fancy) either.
St-Jean-de-Monts has a lot going on at night - all the shops are open selling holiday tat and there are fun fair stalls etc. It wasn't our thing but the kids enjoyed being out and about at night.
Hope that the weather is kind for you. Have a great holiday.