Exfoliation always seems a bit silly to me - the top layers of skin are there to protect the rest of it.
I only exfoliate the bits where it doesn't naturally rub off (my knobbly knees, mostly), and I do that gently with my hands.
And no, my towels aren't full of dead skin. What a weird idea.