Imo the thing to avoid is a situation where you threaten/dish out a harsh punishment, he then responds rudely, then you escalate the punishment, then he responds, then you escalate ad infinitum. The problem with this situation is that in some strange twisted way it's him calling the shots, even if it's in the direction of making his life more uncomfortable.
Sometimes teens almost seem to crave that: they feel hard done by because of hormones, so get some validation for their feelings by engineering a situation where they are hard done by. Don't fall for it. It's not his hormones that should be ruling the roost: it's your experience and level head.
What I have found is that often the quiet stern telling off works just as well as actual punishments. Though it may not seem so, most teens actually want to please you; it's just that they have to try to become independent at the same time. They like to see that you hold the reins and cannot be manipulated.
Secondly, I have found that once you have dished out the punishment that should be it. Do not react to his reaction to the punishment and be tempted to escalate: just stick to your first bid. (Though if you find that it was, in fact, ridiculous, you can rewind the whole situation and start again- as long as you do it calmly and in a way that shows you are in charge- laughing at your own overreaction can actually help here as it shows confidence; this does not mean that he is allowed to forget he behaved badly).
Thirdly, the same argument applies here as to quarrels between spouses: do not generalise! In fact, it applies even more here because teenagers are naturally insecure and prone to wild generalisations anyway, nothing easier than to make them believe that there is no point in trying because they are intrinsically bad people.
So if he was unkind to his brother, do not generalise this into a sign that he is a generally untrustworthy person who cannot be trusted out with his mates: punish him for being unkind to his brother. And leave it there.
When he apologises, accept the apology. Else, where is the point of him ever showing remorse? Teens are dramatic and emotional: if you let him believe there is no point in being sorry because you won't believe it's real, then his response is likely to be that he might as well not bother. That's not the path you want him to go down.