I live here - so honest advice incoming!
You're correct in that there's not loads for the kids to do if it's raining, though we've learned after many years to just ignore the rain and let the kids get muddy - not really an ideal solution for a toddler though.
I'd avoid Matlock if I'm honest - it has the feel of a rundown seaside town, only with less to do. Lots of arcades and "dine in" fish and chip shops. Some reasonably priced antinque shops though (the only time we venture there). The aquarium is okay and very cheap, but very small! More like a pet fish store, with a very large collection of holographic pictures?! Perfectly reasonable for the price though, just don't expect it to last more than 0.5-1hrs. There's rowing boats and sometimes a fun fair, but nothing's "naice".
Heights of Abraham I would, again, avoid. Better caves in Castleton (including one with a boat ride), and the cable cars aren't very good at all. There's a fabulously eccentric man in the junk shop in Castleton who's been there for decades too, if you end up there. Crazy, but definitely worth a browse of his shop if only for the experience! He loves scooters.
Across the path from him is another old boy, who has a gift shop with a cafe above. The trip advisor for his cafe is HILARIOUS, he's even made the Daily Mail as "britains grumpiest shopkeeper". If you have a tenner to burn, go there for a tea served in a cafetière and be forced to buy one of his scones. Just don't say you're a walker or a tourist - he HATES walkers and tourists. There's a new/refurbished visitor centre there that's just opened too.
There's an indoor soft play at the Peak Shopping Village near Chatsworth, haven't been yet but the only one I'm aware of there being in the area, and by all accounts pretty good.
Some bits of Buxton look very nice but it's pretty rough. I think business rates must be very low, because Costa and a few other shops are cheaper and there's no empty shops. I get the feeling they've tried to gentrify it and failed - last time we were in Buxton we watched a homeless man steal an old ladies shopping then proceed to have a piss and smoke crack in the Waitrose/M&S car park. The locals all avoid like the plague. Good bookshop, not worth taking the baghead gauntlet to get there though.
Chatsworth is nice, but expensive and not very child friendly. The farm there, however, has a great play area and animals and is only £6 entry so you don't have to buy a house ticket. Loads of walks round the grounds too, and lots of baby deer and sheep when we drove through the estate this morning! Nice tearooms in Edensor, farm shop is pricey but nice (pick up a Chatsworth leaflet from somewhere, they usually have 10% off farm shop vouchers in them - we keep a stack for our weekly shop).
Peak Wildlife Park is out your way a bit but probably your best bet. Indoor play area and you can go in the enclosures with lots of the animals! Reasonably priced, BOGOF pizzas later in the day and really very good.
Eyam is interesting and has a small museum. Loads of interesting history, plague graves all around the village. The church has a good amount of information about this too. Worth a walk up to Montpesson's well, even if just to look at the field of llamas you pass on the way! Great park in eyam too, it's just a bit hidden (take the path behind the church).
I also second the outdoor pool at Hathersage!
In short, the villages are all lovely, picturesque and full of little hidden gems. The towns are pretty grim (with the exception of Bakewell itself, Old House Museum is good if you're there!) with not enough on offer to make it worth going to them.