Diddledee, I grew up in the suburbs of NYC then lived in NYC for 10 yrs & know it fairly well & continue to spend every moment I can there. I also have a lot of boring opinions about NYC so apologies in advance. Some of the suggestions here are excellent, but would advise against others. For example: I think LI has changed since Janh was there. (Sorry to contradict you, Janh, for I worship your advice on all matters teen.) Nassau County, where most NYC commuters tend to live, is overpopulated & traffic horrendous. It's expensive, too, and dare I say soulless (says the one who escaped with soul intact, I think ).
I'd also worry about Darien, CT, another expensive suburb (like many CT suburbs). The famous line "The Aryan from Darien" is still true, according to an Arab woman I sat next to on the plane. She's found living there with her baby difficult; ethnic homogeneity & conformity are the norm, she says.
If you're after a proper NYC experience, why not stick to the 5 boroughs. Granted, you'll get a shoebox (& no garden) for your money in Manhattan, but how about Brooklyn? You can find some good school districts there, it's ultra hip, you probably won't get a garden but you'll have parks nearby. . .and it's slightly less expensive than Manhattan. It feels urban and self-contained, and it's easily accessibly to Manhattan by subway. If you have no joy in Brooklyn, then try Queens. . .and in Queens you can get a garden.
I'm not that familiar with N. Jersey, but agree it's deffo worth looking into. But again, you might not have the NYC experience to the same extent. (Granted I'm a boring urban curmudgeon.) I also think the London suburbs are totally dif to the NYC suburbs. You might be happy in the suburbs here, but not there IYSWIM.
This goes without saying, but look into schools first. Catchment areas are more clear cut there than here. There are some fab school districts in Manhattan & Brooklyn & (I'm sure) Queens but others that are floundering--so a bit of research will be key.