They're doing some arts/story time things at the National Gallery over easter, also I think at the portrait gallery. My toddler doesn't really care where she is too much as long as she can run around, so I'd even consider the British Museum!
It depends how central you want to be. Kew Gardens is beautiful and has an indoor play area for rainy days, but then you're not really near anything else.
Somerset house is another good place for a hot day, they used to have water fountains splashing from the ground, and you're in the heart of London - not sure if fountains are operating or not.
Also st James' park is beautiful and in the heart of London/Westminster.
I don't rate the zoo, it's expensive, always really crowded and difficult for little one to see the animals. IMO a toddler would be just as happy running around a park and feeding ducks, until it's time for your grown up activity - you can save the money and use it for your lunch!
Or if you go for one of the museums in South Ken you can walk a bit further down to st Lukes gardens, there's a lovely playground there and little garden, plus loads of lovely shops and restaurants, as well as the Saatchi gallery, if you like modern art (and it's free)
I could go on and on, I love London!