Victoria is very well served by public transport - London stations on the lines in from Kent, Victoria and District/Circle line tubes and buses.
My boyfriend lived in West Norwood when I first met him, there's a train line to Victoria but I never got over feeling it was the middle of nowhere in other ways. And it was more than 10 years ago but I didn't think shopping was great, Streatham was marginally better.
Affordable rentals and a safe feel may not go too well together. I live in an area of north London, Tottenham, which rather lacks safe feel but is good for parks and public transport as well as other public services and is still cheaper than many of the gentrified areas near by. Our nearest tube is Seven Sisters on the Victoria line and only takes 20 minutes to Victoria though.
There are lots of Aussies living in the area - Wood Green, Turnpike Lane and Manor House on the Piccadilly Line are more popular with middle class renters and buyers, and are good for shopping of various kinds, and journey time to central London is similar (a very easy switch at Finsbury Park for Victoria).
Find out about Children's Centres near where you're staying now, and look in the nearest library as well. Get the family registered with a GP. There are lots of parent and baby groups that meet in children's centres and libraries, and they're often publicised in these places and in health centres. CCs also provide nursery places, paid for but usually cheaper than private provision and often very high standards, though demand for them is high, especially for under 2s.