There are fireworks in Brockwell Park on Friday night. Can also recommend Belair park and Dulwich park for swings etc for the toddler. Dulwich park has good cafe for reviving cups of tea before or after swings. Also lovely lake, ducks etc.
Dulwich village has nice places for eating and drinking which are toddler friendly. Good garden in the picture gallery for running around in. Horniman museum only a short drive away, as are dinosaur sculptures in Crystal Palace park.
Have a great time!