It wasn’t necessarily so to start with, but it definitely became a part of the story the Empire told to justify its continued existence.
Shall dig if I have time, but the concept of racism as in Black vs White was apparently created in the US after a near successful uprising (I forget where!) where the Black slaves and indentured Irish had bandes together to overthrow the masters. To avoid this in future the poor where segregated according to colour and the White where taught in mass about their superiority. Mucho effective, to let the bottom rung of society have someone else to look down on.
Divide and conquer.
Prior to that slavery was something anyone could experience - it was extremely popular well into the early Middle Age in Europe, where it experienced a lull because of Christianity and that you shouldn’t enslave your brother in Christ. Could have him as serf though…
The western Vikings favoured Irish slaves, but weren’t difficult about enslaving anyone else either. The Bysantine Empire had slaves from all over, but like with the eastern Vikings the Slavs from around the Black Sea were in a precarious situation - they had been for a long time as waves of invaders and traders were drawn to this area.
The trans-Atlantic slave trade was in one way a matter of simplicity for the White European traders: The powerful slaving clans/tribes caught the slaves inland and brought them to the coast ready and waiting. The European traders didn’t have to bother with traipsing through the jungle and fighting to capture anyone themselves. The slaves were a ready commodity (and this bit is truly awful) often held in fairly large camps by the coast - in areas it was difficult to escape from - waiting for the next slaveship. The wait could be fairly long.
I do However agree that after a while a key factor certainly was the racist, White saviour, social Darwinism Theory that was being taught (fed) both europeans and others to perpetuate and justify the colonial powers.
Someone brought up the genocide in Rwanda as an example - I recommend reading up about what the german and belgian colonisers did in Rwanda. How they pitted tribes/peoples against eachother and sowed the seeds of extreme mistrust.
Having said that; divide and conquer is sadly fairly easy and effective if you take existing faultlines in a population and amplify them. Of course most groups would like to tag along to the group perceived as the most powerful, even if it later is shown to be to their own disadvantage.