I think you did the right thing. My ds1 is a simialr age and the opposite to yours from the sounds of it. He's big for his age, very boisterous and lively and loud but never hits or is violent! I'd tell him off if he was! Even the most energetic lively boy should not be allowed to be violent, its not on.
agree with 2.4, Ds is the opposite to your DS, he does like a tumble and gets very excited, but he's never violent - i'd be quite embarrassed/ashamed tbh if he was, and he'd be the one getting removed from the situation.
try a different group, and get both of your confidence back. I tend to find when DS was younger and was picked on a bit, I did as you did, but DS's little face would always look like he didn't know what was happening as he would just want to play.