I wrote one of my law school dissertations on physical chastisement of children under The Children Act, as it then was.
I am shocked at the ‘well it never did us any harm’ comments above. You need to do your research into some of the far reaching psychological damages done to children in years gone by, and still today behind closed doors.
OP, the problem here is that you have role modelled to your 4 year old that it’s okay to hit someone smaller, weaker, and with fewer defences than yourself. Like perhaps your baby sister, the pets, smaller children at nursery…
You have been very honest that you lost your temper and saw what happened. You need a solid plan of steps to take if you ever get close to feeling like that again. It needs to be sufficiently robust so that even if you get to step 10 on that list of coping mechanisms, you know for sure you won’t hit your child.
Everyone is human and some people have more of a temper than others because they are wired that way. You just have to work a bit harder to make sure you can cope in those situations.
Apologise to your son and explain that it is never right to hurt people (or pets) and that it won’t happen again. Tell your partner once you have had this conversation with your son.
All the best for the future.