I have news - everywhere we sit, everything we wear, absolutely EVERYTHING will have dead skin on it. And not always our own.
I'm not a fan of the puff myself but OH is. He sometimes asks me to get him a new one, little does he know he is frequently given the same scrubber back after a go in the washer. I prefer the gloves myself, they last years. And as a PP said, a separate pair for scrubbing veg.