Honestly it's the clothes. So nice we live in a world where mist things are off the peg, anyone who isnt 'fairly average' will find issues. Im not as small as you but am still small, and everything fucking drowns me.
Im not so sure it's vanity sizing, but most stuff now seems to be way oversized compared to the past, and i sometimes wonder if it is cheaper for them to design this way, because more seams and zipped waists and more feminine cuts would be trickier to produce at a mass scale - so i think manufacturers looking for quick profit prefer to churn out stuff that will cover everyone ubiquitously without much variation.
So i think you need a more fitting shirt/blouse. There's just way too much fabric, even if it fits in the shoulders. I have this issue all the time, and even a size 6 silk cami billows out around my torso like a tent. I have no idea why there is so much excess, unnecessary fabric going on 
If you liked vintage you'd have more luck there. People used to tell me higher end and designer were cut smaller but from experience i can vouch that they do not - many of them were even more oversized than the high street.
But i get you, sometimes i dont want the conservative feel of a 'fitted' top, but just wish looser ones were more fluid to my form instead of voluminous. Even now i am struggling to find ANY jumpers that aren't huge, boxy and oversized. When i did find one it was over £200 a Brora. It's depressing.
The same applies to those who are on the larger end of the scale as everything is designed for this homogenous average that possibly doesn't exist, and explains why women in general find clothing never to fit quite right.