We were there last August. Ds was nearly 4 and the other kids we were with were 8 and 5. All loved it.
The blue house is now a very yummy chocolate/coffee shop. However, because ds went in there and had something to eat/drink, Edie McCreddie's house is the one house he says he didn't see: he just says he went to "like Edie McCreddie's house"!
Archie's castle isn't there (as Tortoiseshell says, it's in Berwick), but there is a large pink sandstone hotel just above the town (and next to the yellow house) which as far as ds was concerned was Archie's castle.
It's all fairly compact, so I owuld say it was easier to walk. However, the road up to the yellow house is fairly steep and then you have to go down it and and then up again to get to PC Plum's, Spencer's and Miss Hoolie's houses - which are all on the "upper" level.
We parked at the south end of the harbour (free public parking), stopped in the blue house for coffee/cocolate and cakes (and bought some yummy hand made chocolates) , walked along the waterfront, seeing the red house (from memory a shooting/fishing shop) on the way, up towards the hotel/Archie's castle to see the yellow house (the map is a wee bit misleading on directiosn there) and then back round to see PC Plum's, Spencer's and Miss Hoolie's houses.
I'd agree that the fish and chips from the shop at the end of the pier are very good!
You'll probably find that your kids enjoy the ferry ride as much as the trip to Balamory. Look at for seals!