Three things only, don't have to be child-orientated: I have lived in NY on my own, and spent a few weeks there with a toddler, but if I only had a weekend, I'd not pack too much in and go more for soaking up the atmosphere with a few key sites than keeping to a packed itinerary. More romantic too!
1)At least half a day wandering around Central Park - the world's most romantic and lovely park I think.
Maybe skip breakfast at your hotel, pick up coffee and cream-cheese bagels from a local joint, walk to the park (near the Mall or Bethesda fountain areas particularly great for people/street artist watching, but everywhere is good). Wander down towards the Sheep Meadow, stroll through Central Park Zoo (without paying to go in - it's only interesting for young children), visit the John Lennon memorial Strawberry Fields, walk around the lakes (if it's sunny, a half-hour boat punt here is cheap and not cheesy, plenty of New Yorkers do it).
Finish off where the Park meets the Metropolitan Museum of Art on Fifth Ave on the Upper East side (It's free*, and is one of the world's best galleries packed with masterpieces from the ancient to modern world).
Don't overstretch yourself in the Met, just spend an hour or two in there and visit a couple of things you like; e.g. the Ancient Egyptian section (there's a gorgeous new area that recreates an ancient tomb and lake from Luxor, looking onto the park), then Monet, Van Gogh, Gaugin impressionist favourites and a look at some Warhols etc in the modern art section.
Digest the culture with coffee and lunch or cake in one of their cafes overlooking Central Park at ground level... or cocktails in their hard-to-find but worth it roofbar, which looks down on all of Central Park.
*Many NYC museums operate a 'pay what you can' policy. Do not be ashamed of just giving a quarter or a dollar, it's fine and everyone does it - quite acceptable. So almost free!
- Dinner/night out in the Village or TriBeCa: look up a trendy of-the-moment restaurant, follow with a bar playing live jazz music or similar.
3a) If weather is nice, the Staten Island ferry from the south of Manhattan gives the best views of the famous skyline. Don't get off in Staten Island though - nothing there but a children's zoo - just do a round trip, takes about an hour, free. Follow this with a walk over Brooklyn Bridge at dusk for fabulous views (it takes about 45 minutes - it's a long bridge - but you can catch the subway back at the other side).
3b) If weather not nice, and depending on whether you had your art fix from the Met earlier, go to MoMA (Museum of Modern Art - midtown ten mins south of central park, just off 5th Ave). It's not free - about 20 dollars I think? Wonderful world-leading museum, most stylish building ever, accessible and not overwhelming, with many many famous pieces of modern art.
I would say to avoid Times Square and Broadway - busy, hassley, hectic, noisy, a big traffic island, overwhelming flashing lights and cheapo tourist crap for sale (unless that's your thing). Unless you're going a see a show, that is.
Ground Zero is obviously, a very important place in the world's history and perhaps of personal importance to you, but at present it's a case of peeking between the scaffolding and building site barriers to see the new skyscraper being built. There's no visitors centre or real memorial there yet - and as it can take up to an hour to get there, may end up taking half a day to visit in total. Possibly it's better to save for a future visit, especially as I think it's all due to be reopened soon with a beautiful-looking memorial garden and visitor's centre.
Oh, and I second the posters who recommend going to the Top Of The Rock rather than the top of the Empire State. It's half the price, and you get views onto the Empire State itself, and up of Central Park.
I'm envious, have a wonderful time, and remember most of Manhattan is more walkable than it looks on maps - walk between your sights and you'll soak up more of the city.