Laris, i feel for you, i really do.
Ds was invited to his first ever party since starting school (he started 15 months ago!). He was invited because 'quite a few others couldnt come' weeks after invitations were handed out, to make up the numbers!
I felt quite offended, but he was none the wiser and was so excited that he had been invited to a party, so we went.
Like your ds, he spent most of the party on his own, doing his own thing, didnt eat. I dont tell everyone about his difficulties, i stayed because 'it wasnt worth going all the way home'. Others just see him as being shy and quiet and a bit odd. I did tell a parent when ds was at nursery and she told her child 'ds is ill'
Ds had fun, in his own way. Ds enjoyed the party, in his own way. Even though no one wanted to play with him, ds felt accepted, in his own way.
As long as your ds is getting something from going, i wouldnt worry about everyone else.