I would say that my DS1 was bullied by his best friend at playgroup, aged 3. DS1 was 4. The best friend liked DS1, and they would play well together for a while, but at some point, BF ended up punching Ds1, kicking him, throwing something at him, pushing him over, or similar. Whenever they got together, DS1 ended up hurt - this was mortifying for his mum, as she and I were friends too. It wasn't just Ds1 who suffered at BF's hands, either - most of the other children at playgroup got hit/kicked/pushed/hurt by him. Other mothers were less forgiving and would keep their children away from him.
As they got older, BF and his family move a bit farther away so they didn't see each other so much, but when they moved a bit closer again, we went to see them at their new house. BF by then had a small brother, not quite 3. I had DS2, who was under 2. I watched small brother of BF run at DS2 and shove him so hard he was lifted through the air before he landed on his back, and obviously cried. Friend was again mortified but did nothing really :(
Ds1 eventually got hurt by BF again, so we left to go home.
Since then, I've not been to their house again, but I did invite them both to DS1's last (7th) birthday party because DS1 wanted him there. Small brother didn't take long to hurt DS2, but we thought BF would be ok - wrong! Ds1 was having a great party, no one else caused any problem, but "BF" decided to hit him on the head with a stick. Friend was again mortified, but too late really. We're still friends on FB but I don't suggest meeting up with them, DS1 has pretty much forgotten about his erstwhile "BF" and I don't want to put DS2 in harm's way by letting him near small brother.
There are no signs of SN, just boys who like to be in charge and will do whatever they like to do that (oldest boy now 6, in school, no evidence nor suggestion of SN of any kind).
Some of you may disagree that it was bullying, but I believe it was and I'm glad that DS1 no longer asks to see this "friend" and has other friends at school who are less violently inclined towards him.
Re. the situation of excluding 1 child - no, I wouldn't do this. If there were a particular child I didn't want to invite, then I would reduce the number overall so that at least 1/3 of the class were excluded, not just 1.