It wasn't just the workers in the camps, the camps were most often placed near towns so that people could be used as slave labour in local businesses. That's why the whole rhetoric of people not knowing (perpetuated in media like Boy in the Striped Pyjamas) is so offensive to many, it ignores how easy it is to be pulled into and how hard it is to fight against all encompassing systems built upon oppression.
Why people deny it has filled volumes alongside other conspiracies theories that rely on being the victim of a global plot. There is also the issue that it, while it is taught a lot, it doesn't tend to be taught very well even with all the evidence and places and people working hard to do so and the entire time period and how it affects the world now isn't really done well. For example, Israel has been brought up a few times in the thread, but it is rarely taught that the land they were given was promised to multiple groups to get aid prior to and during the second world war and that is one of the major reasons for the current conflict there (rather than religion that fuels most of the rhetoric). The Jewish people were the second or third group it was promised to, before the rise of Nazism, and this not being taught fuels the holocaust deniers, anti-Semites, and anti-Israel groups as well as causes difficulty in discussing and making progress in the current major ethical problems within Israel when the Holocaust and the creation of the state of Israel are so interlinked.
There is also the major issues that the techniques and philosophy did not originate within Germany, much was gained from other Western countries (particularly the States, the segregation of the time was documented as an inspiration and much of the scientific techniques and practices comes from the States and the colonies of Western countries). I think it is hard for other countries to recognise and discuss the major parts they had to play in this so those parts are almost erased, much like they are in other atrocities to the point that the Holocaust itself is also at risk from the erasure that many in this thread have already heard from others getting more popular.
While Holocausts camps are kept up as a reminder, all the British concentration camps have been all but erased; while we must never forget what happened in the Holocaust, we are so often told that we need to "get over" the atrocities of that happened during colonization and occupations, even when they are within living memory, still have ramifications, and in some places are still happening.
King Leopold II's policies of genocide killed 10 million Congolese a few decades prior to WW2 and yet most people don't even know who he is, let alone put him alongside Hitler, and there was discussion of how little this genocide is covered even within Belgium - yet in Congo it has left a massive wound that still has major affects and those policies were also involved in Rwanda and a major part of the cause of the Rwanda genocide, yet Belgium's influence is rarely touched in the coverage. In the States, Canada, and Australia, more indigenous children are being taken now than in boarding school days and yet this is not just ignored, the mock dress up costumes (which were done to indigenous children during boarding school days to teach them how horrible their culture was and to "kill the savage", which was the entire reason boarding schools were created, they were designed to be genocidal, they planned both for the children to die physically, but to also to kill their cultures if they were to survive) is praised as child's play and used in advertising to invoke the "good old days" when even the boarding schools are in living memory. Being classified as animals is within living memory. Not only is this damaging to the groups involved, it damages the memory of Holocaust as it fuels the belief that these people commonly use that it only taught as much as it is because of who it involves. That needs to be broken by rebuilding many parts of the systems of education and media among others.
There needs to be better education and awareness of the Holocaust, how it fits in history and current events alongside other atrocities and how they affect the world today. There will always be those that need to play the victim in a shady global plot, but they will get less coverage and draw less people in if the wider systems within media, education, law, and so on didn't give them as much fuel for themselves and to convince others with.