I lived in New York City for 10 years, but as a single career oriented person. So, my comments might be very different if I had the experience of having a family in the city.
As bossykate says - what lifestyle do you prefer? Do you live in a city centre now, or in the suburbs? Which is more enjoyable to you? Would this potentially be a long term move, or for a finite period?
FWIW - if you want the full NYC experience, I would try to live in Manhattan. I spent 4 years living in New Jersey - a short bus/Path ride away from Manhattan. But, even though the ride was short in reality, it was a long way psychologically. For some reason, it seemed a huge effort to go into the city when I didn't have to. I was too far away from the city to impulsively/regularly pop in for anything. But, not far away enough that I could enjoy a true suburban life (with the city within striking distance). So, I was kind of in no-man's land.
When I finally moved into Manhattan, I really integrated into the city/culture and began to explore everything on offer. My concern about Brooklyn would be that you also might be not-quite-in-the-city, but also not-in-the-suburbs. But again - please remember that my experience was 10 years ago, and childless. And, I don't know Brooklyn at all, so it may be just the place for you! (Related question: would you need/want a car in Brooklyn? Would you be comfortable parking a car on the street? Many people aren't, and renting a protected parking space can be financially equal to rent on a studio apartment so can impact your budget significantly!)
What will you do while the older children are at school? Will you want/be able to work outside the home? If you're in Manhattan, you could pop out to an exhibition/film/museum etc (taking advantage of the wonderful things on offer), without it turning into an all day expedition.
But, a potential down side to Manhattan living is that I personally didn't know any stay at home mums - most had to/wanted to work for economic and personal satisfaction reasons.
I hope I'm not being nosey when I ask if you will have a budget for private schools? I ask that because my gut instinct is that I would not want to live in Manhattan and send my child to state school - but again, I have no direct knowledge.
My good friends (with a child) live very high up on the upper West Side near Columbia University. They love it. Their son went to a private school called Bank Street that sounds astonishing in it's academics and Pastoral care. I know they would recommend it strongly to anyone.
I had a wonderful ob/gyn on the Upper East side. He is close to retirement age now, but is experienced, kind, and compassionate. I would be happy to pass on his details to you, should you wish to CAT me.
Last tip - Manhattan is hugely expensive, so be sure to figure out how much money you'll have to live on after taxes have been deducted. It seems fairly obvious, but I foolishly didn't do that (at the ripe old age of 25), so that's why I lived in New Jersey for 4 years!!
Anyway - everyone has a different experience, and whatever you decide, it will be a huge (and hopefully fun) adventure.