I am not going to focus my response on whether or not your DS should have been invited to the party. I want to focus on something else.
Simply because I have been where you are right now. I have walked away from school with my child who came out crying without the "golden ticket" to someones party or birthday activity. I actually walked home crying silent tears on his behalf and hugged my DS1 to bits when we arrived home.
I then gave myself an enormous kick up the arse. No parent "owed" my child a party invite, they had simply grown out of whole class parties. I realised that I had to work on my childs social skills and how he interacted with people. Why? Because the majority of HFA children realise at some point that they are different but do not understand why. It also makes them extremely vulnerable and they become even more isolated when their peer group start to do things independently of their parents. Like going to the park and out with their friends etc.,
So, the best advice I can give you is to work on his social skills, broaden his social life and increase his social circle as soon as possible. I learnt the very hard way that if you allow him just to have one or two friends you are never broadening his horizons. My son never had the social skills so that if, for example, his only friend was off school sick or on holiday. He would just sit down with a book at playtime or sit on a bench on his own. It was heartbreaking.
The reality is that at primary school they are often setted for subjects even at primary school age so in just one day he could be sitting next to three or four different children. He won't or may not understand that just sitting next to someone makes them a friend, or a best friend. So, he needs to be taught that you "make" friends and how to do it. He also needs to be taught that sometimes you have to sit next to people you don't particularly like or even know, but that is OK. That sounds harsh but it is true and is good preparation for secondary school where in Year 7 a lot of schools will sit them in alphabetical order. This can cause ASD sufferers enormous problems if they are not expecting it.
I found that Beavers, Cubs or Boys Brigade are brilliant as they all promote manners, good behaviour and always follow the same routine but with varied activities. Also, your son would love doing all the work to earn the badges and he can work his way up to Venture Scouts as a teenager which is when he is really, really going to need a social outlet outside school. They will have experience of Autism and will be patient and understanding but never mollycoddling. That was my biggest mistake, I would mollycoddle him and want to wrap him up in cotton wool. No! Your child will get there socially, but needs instruction and patience because he needs to be taught those things rather than instinctively having the skills.
Also, swimming was very good as the instructions are given in a very clear and concise manner.
I sound really patronising but with ASD and HFA you, the parent, have to teach the tools and skills needed and start them as soon as possible.
Best of luck, Zip x