Firstly, I'm so sorry this is happening to you ajd your child.
The head is right though. If your child is violent in return, then they'll end up in trouble as well.
However, the school are failing in their duty of care to your child.
Schools should have a log of bullying. One of thr best school leaders I know said never believe a school who claims there's no bullying. There will always be bullying between children and teens. What matters is what the school does about it.
Ask to see the bullying logs for your child. All schools should have them. Then write up a summary of your concerns to the head and cc in the school of governors 'for reference'. Explain that you would like to request a meeting to discuss this ongoing situation which leaves you concerned about the welfare of your child.
You have every right to raise this and see a satisfactory resolution.
In terms of keeping him off, I'm not sure. You could request that the bully is specifically nowhere near your child at break/lunch/in class and within eye contact of staff at all times. When we've had nasty bullies like that we have removed all free socialising time and they get breaks/lunch but are kept out of circulation with other children until senior leaders have spoken to the bully's parents, victims parents and we've got a solution
If they won't do soemthing like that then keep him off but ask if the teacher could email you the topics studied/let you pick the worksheets up so he isn't behind. Keep written copies of all of this.
I hope it works out OP.