Good question!
I think it is most probably wrapped up with how much self esteem a particular person has. If someone has a high degree of self worth, then it is more likely they will see the name caller as outrageously wrong, rather than there being any truth in what they have said. For those low in self esteem, they will worry endlessly about whether what they said was true or whether they deserved to be shouted at etc.
I wonder whether much of this comes down to subtle differences in parenting. If we take an example of someone shouting out to a child 'hey fatty, how many pies have you eaten today' the parent with the child could respond in a variety of different ways. Perhaps the extremes are a) parent pretends not to hear, even although the child knows they did - child thinks 'oh god' it is true even my parent thinks so' b) parent might laugh and say 'isn't that the most ridiculous thing to say, you are a picture of health and beauty'. How the parent responds will probably shape the child's view of themselves.
If this continues throughout childhood, it is not at all surprising that the child grows to an adult with differing emotional responses to different things. Of course you do get children who have lived with put downs all their lives who grow up to be wonderfully assertive, carefree people - but these are probably the exception rather than the rule.
It is also worth considering whether the emotionally fragile person is getting something out of being emotionally fragile - it can actually be a form of incredibly controlling behaviour. Looking at another example - if a partner has to go away on business for a week, the emotionally fragile person might be totally crushed by this. However, that person is controlling the life of their partner, keeping their environment as they want it to be. Manipulating those around them to make the choices that the emotionally fragile person 'needs' to keep functioning.
So, in answer to your excellent question - it is very complex