OP, your update about your DD's previous self harm is really important. It tells you that she internalises difficult feelings. I'd defiitely rule out 'shock and awe'. I learnt this the hard way with my own DD. You can have clear boundaries without going on a counter-attack. You can teach your DD important lessons about valuing yourself by taking a more low-key approach, and still not being 'a door mat'. Your DD's history of self harm is a key factor in how you parent her because it it an indicator of how she processes experiences.
(FWIW, Classic, what your brother did must have been terrifying, and I am pretty sure your DNiece 'was never rude again' because she learnt to live in fear. Poor girl.)
I think an awareness of the bigger picture and the importance of preserving the foundations for a good, mutually respectful relationship for years to come are paramount. I try my best to model level-headedness, consistency, patience, fairness, kindness and 'don't-sweat-the-small-stuff' with DD even when she can't do those things herself (she has had a pretty consistently awful year), and it just takes the sting out of whatever behaviours you are experiencing. Let your DD know you hold her in positive regard and don't lose sight of the relationship, keeping an eye on the years ahead. When I look at my group of childhood friends, some of whom (including myself) gave their parents hell, the ones who now enjoy warm and close relationships with their parents are those whose parents avoided authoritarian and punitive parenting strategies, yet remained loving, engaged and responsive.
Young people are so different, and there is a difference between what is 'normal' and what is 'acceptable'. The range of 'normal' is huge! Teen DDs behaving like ours is definitely normal, as are those who do not give their parents much cause for concern. Our lot is perhaps more unfortunate, but certainly not unusual. We just have to work a lot harder for a time.
'Acceptable' isn't a useful descriptor. The behaviour; it is what it is. And we work with it. When we get hung up on notions of 'acceptability' we get snagged into comparison; good and bad, right and wrong. It skims over more useful perspectives. 'Not acceptable!' can also be a trigger for unhelpful knee jerk reactions and punitive interventions which may feel 'right' in the moment, but will strain the relationship, erode trust and won't be supportive longer term.
As parents, we often credit our parenting with the positive expressions of our DC's personalities. We have modelled, we have encouraged, we have set set boundaries and expectations, meted out consequences, taken no nonsense, and "look how well our DC turned out". Until they go off-script and surprise us with behaviour we could never have imagined! I have DDs, close in age, who are polar opposites since hitting their pre/teens. They have been parented exactly the same and have had the same experiences, yet could not be more different. People who meet one, including a range of outside agencies as her behaviour has given such a cause for concern, either pity me or treat me with suspicion, looking for causal and contributing factors to how a previously 'fine' child could have taken such a nose dive. People who meet the other DC congratulate me. I have learnt not to judge parents of struggling children nor think that a 'well adjusted' child is simply a product of good parenting.