Dido, it's perfectly possible to not be scruffy whilst going braless. I've never worn a bra in my life, my mother wasn't really one for taking me and my sisters out to buy bras (I don't know why, I think she found puberty very difficult to speak about, she was from a rather repressed Catholic background), so I never had one as a teenager.
And as I grew older it never really crossed my mind to buy one, I wasn't over-endowed so the lack of one never impeded me in any way.
Here I am at nearly 47, I do own a few bras, I spent quite a lot of money on them a couple of years ago (properly measured, I'm a 34D), I don't even know why I did it, I only wore them a few times, they weren't uncomfortable, but I just felt constrained, so they languish in a drawer. I dress very smartly and certainly don't look like a scruffy slob.