That is one factor. Xenophobia is another.
I don't doubt that most people are probably not racist and that the dislike for Meghan is rooted in other things, many of which are irrational, but racism is definitely a factor for some.
I will never forget reading a comment on the DM that said something like 'Silly Harry. You have fun with the help, not marry them'. And the sad thing was it had hundreds of upvotes.
So yes, it's naive to assume race isn't a factor, even if it's not the case for posters here.