So either it’s a shared pot of money or it’s individual money.
Given you had children, you probably did the lions share of raising them and sacrificed your earning potential at the same time, then I assume it’s a shared pot of money.
Therefore it’s not ‘his’ pension or ‘his’ money. It’s part of the joint household income.
If you are indeed a partnership then he should be taking your needs into account too. If you need to work less or retire and can afford to then that’s a valid request. Otherwise, if you are separate people that share some aspects of life but not finances and not decisions about how you spend time, then when he is old and more frail and you are still fit and able, he can’t expect you to curb your lifestyle to fit with him.
So first you need to decide are you a partnership or two separate individuals with no need to compromise what you want in respect to the other? What is good for the goose is good for the gander.
And if you hadn’t compromised your career may be you would have more savings, a rental income, etc that means you could retire early.
He married someone younger, enjoyed your services as the main parent (assumption but likely) but isn’t keen now on you enjoying the benefits of him retiring and his pension. Surely he’d like you both to be swanning about together?
AND as he’s retired and you are working to maintain a level of income (and presumably supported him in the five years he had no income) he has been doing the lions share of the housekeeping?