I lived there for 4 years before and after kids and only returned to the uk in June. My advice is to get out of downtown. It's pretty uninspiring and you might be forgiven for thinking 'is this it?' Check off the CN Tower, Eaton Centre (shopping) and the Hockey hall of fame if that's your thing and the St Lawrence Market on Front St is lovely too if you are a foodie. Then get out and head to the neighbourhoods. That's where it's happening. In the East you have Greektown, which is a long stretch along the Danforth Avenue starting at Broadview Subway and stretching to Pape Avenue. Lined full of greek eateries plus lots of other options. On Sundays this place is packed with local families having family lunches etc. the greek restos are very welcoming. Try Astoria. Withrow Park is just south of Logan & Danforth, with swings etc. It's a family area. Other 'East' hoods include Leslieville and The Beach. The Beach is funnily enough on the beach of Lake Ontario. Lovely boardwalk and lakeside walking paths, plus the little stretch of Queen St is again full of eating options and shops.
Cabbagetown is also lovely. It's full of original beautiful large Victorian housing. Nice for a wander. It is bordered by Riverdale park to the East, inside that is the free Riverdale Farm which is very kid friendly. Across the valley is Broadview Avenue. Go to the Rooster Cafe for beautiful views across the city and a decent coffee.
Head west and check out Kensington Market, which is an area rather than an actual covered market. Lots of lovely places to head for food, interesting shops etc. It is also bordered by Chinatown (there are two, but the one on Spadina is the biggest). Go for dim sum etc (try Rol San).
The subway is pretty accessible with a buggy. Lots of the stations have lifts, but nowhere near all. But if you are a couple you should be able to handle the steps or escalators together with a buggy. None of the stations are particularly deep underground like in London. The other option is the streetcars, or trams. They have steps but if you are a pair you should be fine - avoid using them in rush hour though they can get packed and might not be fun with a buggy? Buses are also great - they lower for easy buggy access.
I know a few child friendly cafés you can try if you need a welcome environment with a baby. Try Lil' Bean & Green on Queen st east, or Sprouts on Carlaw, or further east at Coxwell/Danforth is Oaks & Acorns. Regents Park Aquatic Centre is a new pool in the Moss Park area of the city which is free to all. Packed at weekends, but they do baby/toddler swim sessions three days a week or at least they used to.
I could go on! Let me know if you have any interests etc in particular or want more info etc. Where are you staying?