I was a size 10-12 beforehand and started feeling horribly bloated from quite early on, to the point where from about 6 weeks onwards I had to dig out my larger 14 clothes from before I lost some weight and put some of my size 10s away for the forseeable future.
However it wasn't really baby bump, more the seizing up of my entire digestive system combined with constant eating due to morning sickness. I probably could have fitted in some of my normal clothes if I tried, but to be honest having anything remotely tight around my stomach made me feel more sick.
I was in maternity clothes from around 13 weeks and both DH and I were starting to notice then that my stomach was getting a bit more rounded and my waist was slowly vanishing. However my friends and colleagues didn't really notice a bump until I was around 16 weeks.
However I did get resized for some maternity bras at 13 weeks and found I'd expanded by four sizes already, so I'm guessing that maybe my chest was providing a bit of a distraction from my stomach, and my stomach didn't seem to be sticking out as far as it was because my breasts were sticking out more than normal, or something??
Anyway, the general public only started to notice at around 20 weeks (as in offering me a seat on the bus). However that's only when I didn't have my baggy coat on, as that seems to still hide everything. I'm just coming up on 22 weeks at the moment.
I'm guessing it's because I was a little cuddly to start with, but my stomach is still more on the squishy side than the solid, stretched belly I would have expected in pregnancy. I still have a slight...crease, across my (less deep than it was) belly button. It's disappearing and everything is slowly getting more solid though, so I'm guessing it's only a matter of time.
But anyway, in answer to your question... It's never too early to start blaming it on the baby! However at present it's probably more the case of the baby is causing the retention of water and pushing out the cuddly bits to make them more obvious, rather than filling the space itself.