These are all completely flexible and therefore slightly meaningless categories.
They are indeed. "Middle age" assumes that you can pinpoint the exact mid-point of your life which of course you can't. But they are arbitrary segments of a life and there will always be people who are outliers because of their health etc.
However, these categories were popularised at a time when people generally speaking didn't live much beyond retirement or the end of child-rearing. The terms old age/senior/OAP etc generally correlated roughly with retirement. Nowadays its possible and even common to be alive for 25 years after you've retired and people's work patterns are far less fixed than they were in the 50s or 60s so a person could retire at 30 or equally continue working into their 80s. And by and large people stay healthier and look younger for longer than they did 50 or 60 years ago.
The age you seem in your 40s or 50s will depend on a huge range of factors including genetics, wealth, bone density, mental stamina, the amount of exercise you take, whether or not you work, what you eat, whether you are happy, substance abuse, smoking etc. It's trite in a way to say you're "as young as you feel" because your underlying health is your underlying health and some people are dealt a bad hand.
But I do find it needlessly negative and a bit spiteful when people pop up to insist that you are actually old at 50. Firstly because how on earth would they know? I'm 53 but I have a very demanding job, I run three times a week, I go out clubbing, I do an extreme sport twice a year. I'm not as young as I was but I sure as hell don't feel like an OAP and I'm not going to be told I'm deluded by someone who doesn't know me because I'm not embracing old age.
But also because your outlook on life does impact how young you feel. My dad was still working, travelling and socialising well into his 80s before dying in his sleep in his late 80s. He didn't live a particularly healthy life: he drank more than was good for him and didn't exercise much. I do firmly believe that his enthusiasm for life and his refusal to be shuffled off into the torpor of retirement was a big factor in the fact he seemed much younger than he was.
It depresses the hell out of me when people rush to embrace an old age mindset when they don't have to.