My DH and I both retired at 59, (DH 5 years earlier than me.)
Once you have no mortgage to pay, no commuting costs, no clothes for work - you tend to find in retirement you just live in jeans/trousers and tops - no lunches, no paying into various work collections etc. then you really need very little to live day to day.
We had quite a good amount of savings and investments. DH has only recently been able to draw his State Pension prior to which he lived from 3 private pensions which covered his day to day outgoings and bills. I won't be able to claim my State Pension for another 3 years but again I have 3 - not huge by any means - private pensions which I was able to draw at 60. We haven't touched our savings so far and we save a further £1000 per month and have a brand new car every 3 years.
We don't go out too often - never have as we prefer to be at home - but we do eat very well and drink a fair amount of wine. 😄. We travel extensively - at least 3 long haul holidays per year and always 4 plus/5* hotels.
Your DH is being silly although sounds as if he has experienced poverty in the past and is scared of going back there. TBH, I was a bit like this but my DH convinced me and several years later I have to agree he was so right and I was worrying unnecessarily.
Sorry to sound morbid but there are no guarantees you will both reach retirement age. It's about getting the balance right! I'm all for saving - as we did - but we have always had nice cars, taken holidays etc. God forbid but what if one of you had a life threatening illness, was left severely disabled etc. which could happen to anyone? Also, bearing in mind the majority of your money came from your inheritance I would certainly be putting my foot down if I were you. Why not suggest seeing an IFA and investing a chunk of your inheritance into a private pension, (Doesn't need to be huge). Job done!! If you do this then you have covered the pension argument and, don't forget, you also have huge equity in your home.
Good luck OP! Hope the above helps in some way.