London is London: it is a collection of micro-areas.
Wherever you live, however you live, within in a very short walking distance there will be people who live a different lifestyle (for better or worse) speak a different language, go to a corner shop stocked with a differnt kind of groceries. Peckham is just like that, as is Streatham , Forest Hill, Catford, Islington, Camden, and....Westminster.
Normal, nice, family people live in all of them.
If you cna't cope with the cheek-by-jowl nature of London's micro-areas, you probably do need to look elsewhere.
Personally, living in an area of s London often considered notorious, and having lived 20 years in Brixton, I have never seen any trouble that was a threat to me or my child at all. I have more often found community support, neighbourly friendliness and security in the streets being always busy.
I would say make sure you don't give yourself the stress of being in an area on the marginal catchment between schools - find somewhere v close to a primary that seems happy, stable and has a good Value Added performance. (No need to stress over the 'holy grail' primaries that peolple pay premium house prices to be near- there are plenty of 'ordinary' good primaries)