When you're a child it's quite simple.
You accept difference AND you acknowlegde it. They accept him and are positive, but they treat him as though he's different because he is. I don't think it's negative, but I understand how it would hurt you.
I am all for celebrating difference (of everyone) and accepting it. Would you prefer it if they treated him as the 'same'... if they did, they would probably therefore treat him badly for his needs?
YANBU, especially regarding inclusion in parties and social events, friendship circles etc., but I think for six years old, the children in your child's peer group sound lovely. Your son will be less different to his peers as they know him more... he will become, simply, him, to them.
Not sure if I've come across right, but whilst I believe in equality in respect and care, equal opportunity and inclusion, I also feel we should acknowledge difference and celebrate it. My school friend WAS different. He was in a wheelchair and didn't share lunch with us (as I remember he didn't 'eat food') and I adored him (and he DID get invited to playdates and parties) but he was different. Over time, we realised we both were, we all were and over time, that gave us all strength of character and a feeling of belonging.
I still do not think YABU though, you're his parent, and obvious differences make you worry... you're not alone though. My son has no special needs, he feels different in that he has different interests and ideals, I worry for him. I shouldn't, because he is accepted and acknowledged.