If you're going to the Geffrye, I found a lovely organic pub just up the street from it. The Fox, it's at 372 Kingsland Road, and they love kids. Weekdays, kids eat free (with a grownup eating, I think? I don't think you can just send in a kid). Anyway, lovely place, no idea what it's doing all the way up there.
Mile End Park has its fair on tomorrow, which is in the Ecology bit (just south of Roman Road, west of Grove), which is a lovely fair, not stalls selling naff plastic stuff, but people demonstrating willow weaving, there's a dog show, but it's people showing off their mutts, not posh dogs. The food is good, it's very mellow and very easy.
There seems to be a lot of classical music on in Tower Hamlets this summer, too. I think most Sunday afternoons, there are bands in the bandstand in Vicky Park (east side of the park, pretty close to the SW corner of the east side).