NannyOgg: "There's too much distance between action and non-invitation for a child this young to see it as a consequence."
To me, that's not the issue. The issue is whether the OP's DD - whose party it is - will be happy to have him around. You might not like to see a four year old child being labelled a bully or a thug, but if you are another four year old child who is being subjected to bullying, thuggish behaviour, it doesn't really matter what you call him, the effect is the same. You're showing consideration for the little boy's rights not to be labelled - well, the birthday girl has the right not to have her life made a misery.
Looking at this pragmatically, what would be the consequence of not inviting him? Will it create bad feeling later on that could rebound on you or your DD? What kind of party are you having - is it one where the children do their own thing, or will there be an adult entertainer who will keep a tight control of proceedings?
fwiw, when my daughter started secondary school, she and a friend decided to have a big party and invite the whole class. There was one very aggressive, violent girl in the class who we were in two minds about inviting. In the end, we thought it better if DD didn't make an enemy of her (in the words of LBJ, better to have her inside the tent pissing out, than outside pissing in). The party went smoothly and she behaved herself.
Not sure it would work the same with a four year old, though.