We took our DTs when they were 3.5 and had a good time but it wasn't the easiest city with kids.
We stayed in apartments that were very central - 60 Urban Balconies (I think) and they were great.
We went to Retiro a few times - really beautiful and plenty of bits to explore. I took the kids boating on the lake (while DH went to see Real Madrid!) and they do have life jackets if you ask! 
We did manage a few hours in the Prado - plan what you want to see and you should be able to nurse the kids around for a bit! The Reina Sofia has some late evening openings so I went there one evening while DH put the kids to bed. They also have free entry on either a Sunday or Monday and I managed to go at that time so that worked well.
We did the open top bus tour which was great. Also enjoyed the Royal Palace. There is also a cable car trip you can do and there is also a viewing tower to go up. Can't remember the names of the last two things but should be relatively easy to google.
There is a nice park kind of behind the royal palace - that might be one of the ones mentioned above - and there are lots of little playgrounds dotted across the city.
We also did the Caixa Forum and there were couple of good interactive art things to do with the kids. The restaurant at the top of the Forum was also really nice - very quiet when we went and I remember the kids enjoying what they chose.
We also had a wonder around the main Mercado - I can't remember what it's called, sorry. And we had churros and bowls of hot chocolate as the very famous churros place (will try to hunt out name!) which we enjoyed even if terrible tourist trap!
I think food/drink were the most challenging thing. Mainly, perhaps, because my image of Madrid is spending the evening sat eating tapas and drinking and that doesn't really happen with kids!! Many places didn't do evening food until 7pm which was a bit late for our two at the time so we had nice lunches and self-catered dinner at our apartment which worked well enough.