Ok, deep breaths. Its really upsetting but don't let your imagination go running away with you and start thinking about his future - cause its not set right? You haven't got that crystal ball. He's not a gang member yet :) He's 4 right?
Its really hard isn't it, when your kids do things that are wrong - but that's where you parent - isn't it?
You know, pulling down trousers etc is something many many children do. Also, lovely children can do really really mean and stupid things. And lovely children also go through hard times when they are annoying, silly, rough, etc etc. Everyone takes their turn parenting 'the kid' that did something wrong. You've had it already. But trust me, it comes and goes and those silly boys often grow into the most lovely wonderful lads.
You should go in to the school and tell them how you feel (fears of bullying) and ask what you can all do to work together. Ask them to explain any class dynamics that are relevant. When did it happen? I'm guessing it was when they were unsupervised right?
Maybe your son needs more supervision at playtime, maybe the whole class do (because he did it with other kids?) Your son is only 4! It is up to the adults around him to help him work through his mistakes. He needs to know you think he is lovely and that we all learn from our mistakes. Ask him how he can make it up to this kid etc- get him to talk to you.
No one (with any merit) will have labelled him the 'rough boy' - there is a tendency to think that all the other parents know your own child. But honestly, they won't. At this age, they will be worried about all the silly things their children do too. Like saying mean things, or being too shy, or not eating food at lunch . . .