Yeah, I wouldn't immediately discount someone because of what they were wearing, but the whole point of a face to face interview is to see the person and see whether they fit.
Also if you had said, I can make that interview however I will be coming from work and therefore dressed appropriately for there then that is just fine. What is appropriate varies obviously.
I don't go to DS's soccer games in my suits, I go in jeans and trainers. Same applies in reverse. If I am interviewing for a corporate position someone turning up in jeans and trainers is immediately at a disadvantage. Because sometimes how a company looks is important to their business as well.