Yep hourglass here as well: as much as you would like to throw on a T shirt and cardi and walk out the door, it ain't gonna happen sorry! Learnt that the hard way 
Is your mind set on a cardigan? You can find other pieces on top to keep warm say a wooly soft scarf? But the tops immediately next to your skin SHOULD be structured (even if it is a casual look you are aiming for)
Maybe a ribbed thick vest with a light shiny jumper on top? Jeans or denim pencil skirts. And make the outfit casual with trainers and other accessories...