Yes, definitely.
I sat on an interview panel for an NHS admin post. One girl turned up in a cropped white shirt and a black denim mini skirt with a (deliberately) frayed hem. I did judge. You should always wear, as a minimum, something at least as smart as you would be expected to wear in the role, preferably smarter.
She didn't get the job, but thankfully because she wasn't qualified, not because of the way she dressed.