We have always talked of treating one and another with mutual respect, and respecting personal boundaries, so doing something to somebody else when they have asked you to stop, has always been something that we have discussed as not appropriate. Since they were quite young any fighting -playing or serious, would have been called to a halt, if someone said "Please stop" Or "No please don't so that" so it was difficult when they went to school and found there were people who didn't respect those boundaries.
We had loads of discussions around avoiding such situations, but only hitting back in real self defence situations (and that we would support them in those circumstances)
We have continued to live with the rules of mutual respect.
If one of my children were to hit me, I'd wonder what on earth had got them to such a state of anger or frustration that they would feel the need to hurt me. So I'd explain that the action of hitting me had really hurt, and ask what I had done or said that would cause such an action.
I'd ask if there was anything any of us could do to resolve, manage or release that anger or frustration? We'd try and resolve the conflict in a consensus way, that everyone felt was satisfactory.
I haven't experienced any of my children hitting me, but I am pretty sure if such a thing were to happen, or if they used language abusively, allowing them a place to express what ever frustration or anger they felt, without resorting to abuse of any kind,would help. We talk about it being normal to feel frustration, but we try and help the children learn coping strategies, as abusing someone else is never right.
Being taken seriously, being listened to respectfully, as an equal, does help I've found.
I've seen parents of autonomously home educated children who have Aspergers and have had a explosion of feelings (and I fully accept that not all people with Aspergers behave in this way) that they couldn't control, deal successfully in this way too.
I suppose if it continued to happen, I would deal with it, as I did with my middle child, who found the transition through teenage the most challenging, and leave the room. I guess I'm with Ommward and Fiona, I simply don't engage with behaviour such as you are describing. Once the immediate behaviour had stopped, I'd once again try and help the child identify the problem/frustration and see if it could be resolved.
I've seen children come to our local HE meetings, recently out of school and obviously previously bullied, decide that they will be the top dog in this new group-and do some thing quite violent or challenging in some other way and uncalled for.
The HE teens in the group have always been able to deal with this quietly without calling on adults, by simply saying "You don't have to behave like that here mate, it's not school, people will listen to you" and then moving on to something else, leaving the person with the option of joining in.
As a family we've always been able to use such terms as "Can we start again please" as a way back from frustrating behaviour. That could apply equally if I was being unreasonable not that I ever am of course
"I'm sorry I shouted" works well, and anyone child or adult in our family, is able to point out to me someone else "Did you know you are shouting?
So mutual respect, is the only rule in our house (apart from the one about the drinks on furniture and no drinking my real ale)