It's up to you what you do and its a difficult decision, but speaking from my own experience I was in a similar predicament and it all went wrong for us.
We rented privately and as house prices soared we couldn't afford to buy. We got offered a council house in an area we didn't think was too bad. Not an estate...just a council road and there were plenty of lovely residents, but unfortunately we lived next door to an awful family and dead opposite another equally as dreadful family.
We kept ourselves to ourselves, but these people liked everybody to know their business. It was like the Jeremy Kyle show everyday with regular arguments in the street that would go on for hours, usually between these families and drug dealers. There were always big gangs of youths congregating outside, drinking, swearing etc.
Within time these people started to go out of their way to annoy us (perhaps as they got no attention from us) and I had my car damaged 4 times.
The 15 year old boy who lived next door would regularly urinate up against our wall and would flash his genitals to my children who were very young at the time.
We also had nights on end of loud hard banging house music which frightened our children and meant that they couldn't sleep.
I complained to the council, environmental health, the police etc. but nobody could offer any help. We had back up from other residents who were experiencing the same difficulties but there was nothing any of us could do. Infact the council rewarded one of the scum families with a costly loft conversion.
Relations between me and my partner went downhill as we would be at loggerheads as how to deal with these people. My partner would want to go and confront the youths which I worried about because of revenge attacks or worse.
Eventually things panned out well and I had the opportunity to move.
The stress has all gone now and its lovely to lead a normal life.
However saying all this, I lived in a street with 80 houses and 78 houses had lovely residents so there was an upside. In fact they were some of the most decent people I have ever met and were a good support network. I felt guilty leaving them behind.
Can you view this street, perhaps go to the street at different times of day and have a peek at what is going on down there.