I'm afraid most of my knowledge is south of the river but:
The Science Museum apparently has a brilliant section for real littlies with lots of buttons to press etc. Agree he might be too young for many museums though.
Battersea Park has a brilliant little zoo with monkeys, limas, farmyard animals etc, and a nice play area. And of course it's a nice park to walk in too.
Have heard good things about the Museum of London (one in the Barbican which I have been too and is good), but another has opened somewhere in east London (Bethnal Green??).
Mile End Park is supposed to be fab after the regeneration a few years ago?
There's a sweet little city farm at Vauxhall, very near the station.
Walk along the river on the South Bank is great (keep going east from the London Eye/aquarium past tate modern, gabriels wharf etc).
My ds is 2 at the end of April so I'll be on the lookout for places to go this summer too!