They're definitely not too young for Tate Modern- kids seem to really respond to it, ime- big cavernous building to explore and many of the temporary big installation pieces are designed to be interactive. Wandering along the south bank is one of our favourite family activities and can be combined with a trip to TM, Jubilee Gardens playground, watching the skateboarders and climbing on the parabolic curves sculptures.
The cloisters in Westminster Abbey are nice to run around.
The British museum has some very child friendly bits- mummies especially come to mind.
Horniman museum in SE London is definitely worth a trip- the Aquarium is good for that age group.
Greenwich is a fantastic day trip- go by DLR or river for maximum excitement and take in a selection of Maritime museum, Queens House,
Observatory, river trip/walk, Greenwich market, pie and mash, lovely park.
Even further East, Woolwich ferry is a nice free trip and there's a good railway museum round there
Battersea Park is wonderful- zoo isn't free but is cheapish, adventure playgrounds to suit all ages, cafe, lots of space.
And Corams Fields in Russell Square is a wonderful place to take kids of that age.
Will try and think of some more ideas and post later.