I've been thinking about this, and I'm having a little bit of a rethink.
In large classes, it's fine to invite less than half, if you want to miss someone out.
But what happens if you are in a small classroom? If there are only a few children of your DC's gender?
I've been faced with a difficult child (child difficult but manageable, but mother difficult and incredibly unmanageable), and one time I only invited a few, mixed gender, to the birthday activity. The other time I invited all the boys, but he didn't come, for which I was relieved.
That's a difficult child, who would upset and hurt my child relatively frequently but was ok when monitored closely by the school (they would occasionally get lax and the behaviour would immediately start again). But that was behavioural issues exacerbated by problematic parenting (so having the parent there wouldn't help, in fact it would actually hinder and the child would behave worse knowing that no one would be able to say anything to them without the mother losing the plot at them).
If it was a child that deliberately bullied my child to the point where my child was in constant tears, didn't want to go to school etc. then I would stop being PC about it and wouldn't invite them. If it was a small class and there weren't other children that I didn't want to invite then they would be the only boy/child that wouldn't be invited.
If someone took umbrage with that (parent of child, or school) I would ask what actions they were taking to stop this behaviour that was causing my child to suffer.