I think it is very valid to chose a school where you think the chances of your child being accepted, making friends and generally having a happy time are higher. Especially if they have Aspergers, and find making friends difficult. And if you yourself have been badly bullied you will know first hand the consequences of being ostracized. However I would be looking more at the diversity in the school, as IME the more diversity the more likelihood of acceptance of difference.
I was sent to a Catholic primary, and was bullied. My mother (who almost certainly did not want to send me to a Catholic school in the first place, but was pushed into it as part of marrying a Catholic) sent us there because it was a good school, it got good results, and had lots of excellent extra curricular stuff going on. But what she didn't notice, or perhaps didn't think through was that all the children were very local and attended the same church (we came from a little further away and went to a different church); they were predominantly working class (we were really quite posh, and sounded it); and they were also mostly Irish (we aren't). We stood out like sore thumbs.
It was not a good experience for any of us, and for me and my big sister affected us quite deeply in that we both felt we were incapable of making friends, something we didn't overcome until university. My middle sister on the other hand was very happy, but she is much more outgoing than us. My mother was fairly oblivious of this, and when I told her, many years later, she said she had also found it very difficult to make friends with any of the other mothers (she can be a little oblivious to the fact that her rather patrician attitude and accent can be a little intimidating).
My husband taught in a school in the East End where 95% of the children came from Bangladeshi families. The 5% of children who were not did have a difficult time.
My kids were recently at a school where 50-60% of the children were Korean. ds said it reduced the amount of friends he could make, but actually his best friend was one of the Korean kids, so I'm not sure how true that was. However it certainly was true that the Korean parents did not mix with the non-Korean parents (quite a diverse mixture in that group too, as we lived in Queens, NYC, one of the most diverse places in the world). That did make it difficult to try and arrange playdates, especially as the Korean children mostly were picked up by grandparents, who did not speak English.
On the other hand my step-niece seemed to thrive at a school were she really was the only black child (and it was a big school). She certainly had a lot of friends.
So it can be, but isn't always, an issue to be different, and if you are concerned that your child appears different anyway, of course you are going to worry. That's perfectly legitimate. I do think the OP needs to look much wider than black-white, as there are so many other factors at work. Visiting the schools is the only way to judge whether your child will be welcomed there, and results are not the most important factor (assuming they are both OK).