Hi
We went last Easter with a 9 and 7 year old for 4 days - not long enough really. It's a fantastic city, make sure you wear some comfortable walking shoes... We loved it and want to go back again.
If you like walking, beautiful architecture and eating then this is the city for you.
Did a fab walking tour with OGO Tours, free you just give the guide a tip at the end. We were lucky it was just our family in the English group and she asked us what we wanted to see. Took us to lots of great food places that locals go to whilst giving us a bit of history. It was 2 hrs and I thought our kids would get bored but they loved it!
The palace is good for a few hours, especially if it is raining. They have an amazing armoury which our kids enjoyed and they loved wandering around looking at different stuff but sadly not long enough for us adults to get a really good look around.
We stayed in the Cortes area which was very historical and a short walk from the Prado, but we didn't go as our kids didn't want to see a museum. We took them to the botanical gardens which is opposite but to be honest I don't think it was really worth the entrance fee, nothing really for the kids.
Go to the park at the top of the hill, name evades me, it's huge and you can easily spend the day there. Great little playgrounds dotted about, a lake where you can hire little boats, cafes, a glasshouse with another little lake next to it that had lots of turtles in it. Great to take a picnic or you can get stuff at the cafes there but we didn't find the food that great for what we paid, better for an Icecream or coffee. Great places for kids to run around and burn off some excess energy.
If you have any other questions, ask away. Hope this is helpful.