I don't see anyone arguing for further population growth - rather, the argument is for stabilizing the current population or for a more managed decline (that is less likely to result in a sudden economic catastrophe or societal collapse and the suffering that may bring).
Incidentally, while it would likely be a boon (at least, initially) for most other species if humanity disappeared, largely because of our environmental impact, I don't think humanity's other moral failings you list (slavery, genocide, torture, manipulation and oppression) are themselves good reason to think the world would be better off without us.
There are many other species who do things that would be considered morally terrible, we just dont hold them to our standards (for obvious reasons).
If, for example, you could run a poll of rodents and birds (well, chickens and turkeys excepted), I imagine they'd sooner get rid of all forms of cat before they got rid of humans.
(Slightly complicated by our role in spreading cats around the world, but you get the point).
There are, too, some foreseeable circumstances in which humans might be the only species capable of saving the planet (for example, from the threat of a catastrophic meteor strike).
I'm also far from convinced that lobbying for humanity's extinction (not least as being the only possible way forward) represents the moral high ground you think it does.