Ok, the scarf thing was a bit of a red herring. If you don't want to say where your scarf is from, just say 'oh I can't remember'. But usually it is a compliment (I never ask where things are from, but I do admire things I think are pretty or unusual on a colleague).
It is also true that workplaces can be pretty gossipy, so if that is bothering you, keep away from the main offenders (be polite, don't seek them out so they think you love a gossip) and look a bit vague when they start up, just don't join in. And as I say, find one or two really nice colleagues and ask them for a coffee.
Your original question was about why you think so much of what others say, it's hard if you feel you are the object of discussion, but remember, it's likely to be very fleeting, you are interesting, but not THAT interesting to anyone else (who is thinking about themselves much more than about you). The tips on CBT will help enormously in putting these 'oh god, what are they saying about me' thoughts into perspective.
But you can't change other people's behaviour, especially gossiping in an office, just shield yourself from it and carry on your own far more interesting life.