It's been a few years (ok, try decade) since I was in that area, last time I stayed we were in Chapel Stile.
I recall there is a slate mine in the langdales, with a craft shop in one of the villages (Elterwater?). Some fairly nice bits there.
Sarah Nelsons Gingerbread shop in Grasmere is yummy!
A lot of the nicer places are a walk away and a bit of a struggle if you are carrying little ones. Stickle Tarn is a nice bit of water to dip your toes in, but it's the wrong side of a 370 meter climb. Be warned also, most of the bodies of water in the Lakes are much colder than you might think.
So to visit try...
Grasmere Village (including the gingerbread shop)
Buttermere lake and village. Its a fairly nice walk around the lake too (although too far for the 5 year old to make it on his own).
Ambleside, can be a bit touristy, but worth at least one visit
A trip into Borrowdale might be fun, a couple of smallish villages up the top and the Bowder Stone near Grange.
If you feel a little adventurous, you might try Wasdale Head. Not really much there other than an Inn and the path up to Scafel. But definitely a interesting landscape.
The Lakeside to Haverthwaite steam railway might be a fun trip.
Don't know of any castles.