I got moved by my housing association from a lovely little maisonette in a beautiful quiet leafy street (ours was the only converted house into flats, all others were big detached properties that most people living there had been in for decades. Beautiful nature walks, lake with lots of kids facilities nearby, huge town centre nearby. Lovely park and playing field right behind, then a huge nature park just down the hill.
We were moved because somebody moved into the small flat below us, much if the behaviour you describe ensued. The property turned to squalor, infested with rats and mice and stinking, which was awful as it was an adjoining property and not a proper separate one. Strange people would come and go at all hours of the day and night, and you could hear violent encounters often. We shared a doorstep and sometimes you would have to wait to go out, because there was an angry wasted person hanging round outside, or an armed police raid/police kicking the door down.
We were moved into a block of flats barely a mile away but it was like a different world. Admittedly, being only a couple of estates over, we were still close to a lot of the facilities mentioned before, and even closer (way too close) to the town centre. It is so much fucking worse. If our last place was one bad apple spoiling a pretty tree, this place was just a bunch of rotten festering mulch.
The downstairs neighbour, despite being loud, antisocial, awful and annoying, made it their literal life mission to make our life hell, to make us feel as unwelcome and unable to live in our home as possible. Most other neighbours in the block are fortunately reasonable, but we have plenty of times left to take the kids to school and had to walk past someone who is off their face, screaming and vomiting, or having loud inebriated fights, or shooting up in the lovely communal garden (that we were told is unsafe to let our kids play in, so is a complete waste), or lying wasted right outside of our door, leaving piss and condoms and drug paraphernalia behind.
Honestly, living here has broken me and changed me forever as a person. I don't know how people cope with living in environments where they feel constantly on edge and threatened. Flats are so so much worse because there is so much more friction, and so much less privacy.
OP, do you like it where you live? I do think that's all that matters; there are different kinds of rough that are more cultural and less threatening.