I have been twice, both since 9/11.
First time was for a week in May 2009. We stayed on Broad Channel and travelled in by subway each day. We went with friends and had three children with us, two aged 7y and one aged almost 4y.
Second time we went iin July 2010, for 5 days and stayed in a hotel in Manhattan. We went with PILs and took DD who was then 8y.
Both times we found NYC very friendly and we really enjoyed our visits. I would definitely return. TBH we found it far friendly than any other big city we've been too, much friendlier than London. Nothing was too much trouble. If you stoppd with a map open someone would ask if they could help, on the subway people man offered their seats to us every single time, a woman went completely out of her way to show us which station we needed, another man directed us to a preferred restaurant when he saw us looking at an alternative (he was right!), etc. I never once felt at risk or even alarmed.
Both times we did go to Ground Zero, although at that point the memorial wasn't yet finished. We saw it being built and saw the temporary memorials around the place and in the nearby church. Even around the site there was no worries regarding security.
I would definitely recommend a trip!