Dear JugglingMama
Lots to consider. Will you be in the location long term? If so you'll need to consider secondary schools. Are you keen to enjoy a similar urban vibe to that which you can enjoy in Brighton, or are you keener or something more suburban, or rural?
Cards on the table. I live in Ashtead and have been there for 14 years and was in Epsom for 5 years before that. I work in Sutton. Here's my advice, for what it's worth.
If you'd really like to live in the country or with very easy access to the country, go for Dorking. Nice town, right on the North Downs and Surrey Hills, lovely for walking and cycling with the kids. Far enough away from 11+ boroughs for very few to bother so the main secondary (Ashcombe) well regarded and well used.
If you'd like to feel you are really part of London then go for Wimbledon. Property expensive and I'm not sure what the non-selective secondaries are like. Very nice though.
Banstead. Hmm. I've got a bit of a thing about Banstead. Banstead village a nice little shopping centre - but Banstead station a good hike from this centre. The houses near the station handsome enough, but it's just acres of suburb without a real centre of its own. Handy for big Asda! And no trains run from Banstead on a Sunday. The secondary school serving Banstead does not seem to be well liked - but I have no personal experience of it so check the latest OFSTED.
Both Ashtead and Epsom nice places to live, with good schools at primary and secondary level, easy access to both London and the countryside, a good range of housing of all sorts. I wouldn't bother with Epsom Downs - a bloody nuisance of a place to live, especially if you are trying to get to Sutton by train. With a toddler you want to be within a buggy push of anywhere you want to go, and this is certainly the everyday reality living in either Epsom or Ashtead.
Hope this helps. Please PM me if you want more info.