I can so relate to this. Primary school labelled me gifted at a young age which I'm not, I'm smart, but in much the same way as loads of other people, not 'university at 12' fodder or anything like that. Unfortunately my dad got very excited about my 'potential' once the school had stuck that label on me, and although he and my mum used to tell me it was enough to do my best, so many contradicting things were said that made it clear to me I was supposed to be head and shoulders above everyone else. I was often pushed to read improving books etc rather than just playing and doing normal kid stuff.
There was also an assumption I'd fall instantly into a high-flying (and high-earning) career after uni just because I had a degree, and my dad wasn't quite able to hide his disappointment when it didn't really pan out like that, so I ended up feeling like an underachiever. The reasons he gave for my apparently being 'held back in life' (which wasn't how I felt) were my 'shyness' and my weight. Yes, I can be shy, but I've never been anywhere near as shy as I was perceived to be. I've held teaching jobs, I do amateur dramatics, things I was warned away from at the beginning because I was 'too shy' for them. And although I have had my struggles weight-wise, when I look back on old photos I realise that for a lot of the time when I was being told I was fat, I ... wasn't fat.
Unfortunately, in addition to being too fat and too shy, I was also frequently labelled difficult, awkward, argumentative, moody and bad-tempered which looking back I actually don't think I was any more than the next child/young person. Apparently I'd never have any friends, never find anyone to marry me etc. They were wrong on both counts. They did love me despite how it sounds from this post, but it was a very controlling sort of love; they both had very authoritarian upbringings and I think they had assumed they could just mould a child of their own into a meek and docile little thing (probably why I was encouraged to see myself as shy) and it was clear to me that it had been a disappointment to them that that hadn't panned out. I was an only child which didn't help, because I grew up hearing these labels only being applied to myself and didn't have any perspective (in the form of how they might have treated my siblings) to tell me I wasn't universally disagreeable and unlikeable as they seemed to think.
Tbh all of this has been very hard to shake, and it's only in midlife that I've really had any success at unpicking/debunking any of this in my own mind. I still don't have the confidence I should have, though, because lacking it in the past drew me into unhelpful friendships and relationships which perpetuated the vicious circle.