Oh dear, this isn't going well.
OP, I can see that you are trying to be kind.
As a mum of three NT kids and one ASD kid, who usually throws big class parties, and a teacher, I would like to offer you my unjudgemental thoughts.
I would always invite the whole
class or a select group. Excluding one child only doesn't seem right. DD had this dilemma, we talked it through and so she decided to make her party girls only (the kid she didn't want to invite was a boy). It was 100% her decision, but I could not, in all honesty, allow her to invite everyone in the class, except one kid.
Ask the teacher. She will have insights to social dynamics of the class and might be able to offer guidance on how to manage your apprehension.
Avoid 'armchair diagnosis' or general ASD judgements. Sadly, these days saying someone is on the spectrum seems to be the acceptable go to phrase for describing kids whose behaviour might be a bit off, and it is offensive to those of us who have to deal with this incredibly complicated and complex disability. I agree with PP that referring to SN is much less specific and more appropriate (and I absolutely agree with your point that if you hadn't mentioned SN at all, lots of people would have brought it up).
As the mother of a child who is slightly different, both he and I would be delighted to receive a party invite. DS1 gets about a third of what DD gets invitation wise. Equally, I would have no problem staying to help. As a working mum, it would be a chance to get to know some of the kids!
Good luck. The fact that you posted on here shows that you are thoughtful and kind, and trying to approach this in the best way, with a kind manner. I am sure the party will be a roaring success!