Seeker, I don't like kids' parties much and have tried to keep ds's parties simple and small. It's not as though ds doesn't know the kids I did invite, he plays with them a lot (because I'm friends with their mums) and they had a great time in the paddling pool, which I couldn't have done if I'd had a big party with all the kids from nursery.
When he's old enough to draw up a guest list then I will invite whomever he wants.