I'd just wash what was I was going use just before I used it if it looked a bit greasy or dirty. Otherwise - well, what's going to happen? Unless everyone in the house before you was infectious with some dreadful disease that can endure on glass, metal and crockery, you'll probably be fine anyway.
I think it's just the idea of everything not being clean that people don't like.