The history is quite complicated - interesting and sad, but complicated. As you know, the country that makes up present day Nigeria is made of different ethnic groups and the remains of different empires (kind of like many modern European countries are). You are correct that there was slavery in those lands long before the transatlantic slave trade, just as there has been slavery in almost all human societies. The systems of ‘domestic’ slavery in the societies along the west african coast and the rules that governed this varied very much depending on the society that was practicing it - from debt bondage to hard labour as a form of criminal punishment to religious slavery to the plantation slavery that we are more familiar with. However, they were all very distinct from the transatlantic slave trade that was conducted with Europeans, which was slavery on an industrialised scale purely for economic reasons. All slavery is wrong, but arguably the industrialisation and racialisation of transatlantic slavery brought about its own unique form of dehumanisation. All slavery is evil and a crime against humanity. However, enslaved people in those societies were still part of society and had some civic rights. They were able to earn/buy their freedom in many cases inherit and have families, their owners had some duties towards them and there were some checks and balances against their abuse. This was not the case in transatlantic slavery, where the racialised system meant that enslaved people were seen as subhuman and treated as animals.
(I am not trying to justify slavery in those societies by the way. Like in the Inca and other mesoamerican civilisation, there is evidence of human sacrifice in religious ceremonies and some of those sacrificial victims would have been enslaved people. Like I said, slavery is evil and a crime against humanity.)
The reason I don’t think the Osu caste system is comparable in this situation is that although it is called slavery, it was more a system of ostracism. The Osu were people who were considered ‘slaves of the gods’ because they had forfeit their lives to the gods in exchange for avoiding worse fates like execution in some circumstances or even sale into transatlantic slavery, either due to taboos, or in some cases by descent (the descendant of an Osu inherited this status). This made them outcasts from ‘human society’ and meant they were marginalised and unable to relate with others. It was a cruel system but is more comparable to the system of ‘untouchables’ in the Hindu caste system than to the system of domestic slavery. If you wanted to look at systems of modern slavery, examples abound within modern Nigeria - the treatment of some domestic workers for example, including child slavery (which is illegal in Nigeria but still exists), forced labour of boys and girls, people trafficking and sexual slavery, the kidnappings of schoolchildren in the north and their enslavement by bandit groups with girls being forced into marriage and boys forced into violence and manual labour (the terrorist group Boko Haram being an example that many people are aware of globally)…there is a governmental response but a lot still needs to be done.
Coming back to your point about elites versus enslaved - I am sure you know that history has its own humour. British colonialism upended many traditional systems so a lot of the people who are elite today would have been the descendants of people who were enslaved, particularly in Lagos. This is because there were enslaved people who came back from the Americas (for example the Afro Brazilian population in Lagos and the Saro community in Lagos who are descendants of returnees who were liberated by the British navy during the time of the abolition of the slave trade and returned to Sierra Leone - which was founded by the British for the freed slaves - and made their way back home). Some of Nigeria’s founding fathers were descendants of such returnees. Herbert Macaulay is an example of this. Asides from this, the people who did well in the new colony were those who embraced western education and people who were marginalised in the old systems such as enslaved people were more likely to do this - to accept the new religion of Christianity, attend western schools and thus be placed to have better positions in the changing society. So while there will be people in the present day Nigerian elite descended from people who owned and traded in slaves, there will also be many in the elite who are descended from enslaved people.