Yes, I agree. Present-giving is part and parcel of children's parties - excuse the pun.
If you want alternatives to yet more toys, how about suggesting to the givers that their child makes something very simple for your son - obviously with some adult help - ie a collage picture, some home made sweets, a seedling planted in a pot etc. If you have a garden, perhaps you could set a space aside for your son and suggest seeds and plants as gifts?
Alternatively, if you don't think this is too mercenary, you could suggest a small money gift, so that you can put it in a savings account or use it on a bigger toy later in the year.
Whatever you do, I think it's a good idea to suggest a few alternatives, then those who don't have much time to shop for a toy can feel guilt-free about popping some money in an envelope, and those who want to make something can do so, while the traditionalists can still get your little boy a small toy gift.
IME, once the party is in full swing, who gave what is soon forgotten!