No it will be fine. It is a 3 hour drive if traffic is ok. The only real hot spot you may meet is the Nantes periferique, but that rarely stops completly, so may add 30 mins to your journey. The motorway after Nantes rarely has problems unless there is roadworks (which never happen in August ). Depending on where you stay in the Vendée the coast roads can get busy, but if you stay somewhere like Fontenay le Comte, just off the Motorway then that is never an issue.
If you take the overnight boat, then you should clear the port by 8 am, and should be easily at your destination by lunchtime.
Most French people take their holidays in the first two weeks of August, so the Saturday around the 15th is when everyone goes back home. Roads around tourist spots are often hell (family live in Marseille, we don't go near the roads that weekend <shudders> ).
But as frenchfancy says, it's much worse going north. The Saturday two weeks before is worse going south. But as you're avoiding that one it should be ok. Especially as you'll be out of the port early.