I'm sure it will stick in your craw, but might it be worth investigating exactly what this mother (and, I suppose, the 'core' of other parents who have excluded your son in the past) is upset about with regards to your son's behaviour?
You said that he can be rude - how does this manifest itself?
The calling out and disrupting lessons thing - of course he finds it very hard to control and presumably you and his teacher are working on strategies for this - but other children do find this kind of disruption very irritating (and it can be demoralising for quiet, shy children who never get to answer a question because the answer is always shouted out).
The idea that someone or something is "disrupting my child's education" is like a red rag to a bull for many parents, who (understandably, just as you do) want their child to have the very best and not have (in their view) all the teacher's time taken up by one demanding pupil.
At the moment they may just see the situation being one in which your son gets all the time and attention and their children get nothing yet they are expected to "put up or shut up".
If you can acknowledge that you understand your son's presence in the class impacts (sometimes negatively) on their children, this may go some way towards a softening of their attitude.
This might be a way in to explaining what you, the teacher (and of course, your poor son who sounds as though he is trying so hard) are doing to address these issues, and how they can explain it to their children and encourage them to support him.
Basically, reassuring them that you are not in fact expecting them to "put up or shut up", but you are working very hard to make things work.
This may then also lead to much more positive stuff about your son - you say he creates imaginative games etc that the others really enjoy - it's probably useful for the parents to be reminded of this, so that your son isn't seen as just a problem that needs solving but also an asset to the class and their children's lives.
All the above sounds as though I am agreeing with what these parents are doing and am attacking and blaming you and your son. Please don't think that, I'm really really not, I'm just wondering if seeing it from their point of view and acknowledging it might be a way to open up some dialogue to improve the situation.