Il de Ré is indeed lovely. My advice would be to go the last week in June or the last week in August. Between the 2 it is really really busy.
For kids under 4 I find a place with a private pool away from the beach with a couple of beach days inbetween more relaxing though. I never relax on a beach with little ones.