Of course you should allow him to have access to his brother, larry! He has to learn to socialise with other people and to develop a good sibling relationship. Keeping them apart would not be the best way to avoid occasional lapses in toddler behaviour, either.
But as their parent, you know them best and you can probably spot a situation that may escalate into violence - that's where distraction techniques etc. are good to try and divert things before it gets to the point when one brother attacks the other. It's not always possible to get in there in time, but when it is that seems a good way of avoiding problems before things get out of hand.
"Kind but firm" is the mantra in the Positive Discipline books. The idea is that you don't scream at a child and treat them roughly when removing them from a situation where they have misbehaved. Humiliating them and upsetting them is not the idea.
Modelling behaviour that rough physical treatment and screaming and yelling is OK will bring about its own problems. Have you heard of "mirror neurons"? It's about people learning things by mimicking what they see/experience.
But we all lose our tempers sometimes, it's not easy to remain patient with a toddler pushing away. It's not the end of the world. But it's better to try not to.
I'm not anti-punishment per se. I'm anti-punishment because I'm not convinced it's necessarily the most effective approach. But I admit, that will also depend on the individual child's temperament.