There's calculators online that will tell you whether it's worth paying the ERC. I suspect not as it's equivalent to about £150 pm over the remaining fixed term.
You need to compare with the difference between interest earned after tax and the cost of the mortgage. I suspect this will be smaller than the ERC.
I would pay off as much as I can without penalty and plan to pay the rest off when you can do so without paying the ERC.
look at your ERC terms carefully, what can you pay off and when. Sometimes if you pay off within the last few months of the deal, the penalty won't apply.
For other spending (home improvements, holidays) I'd use savings and possibly 0% credit cards, as would be cheaper than keeping a mortgage, but that depends on what rate your mortgage is and other things like what you can afford to save.
As your income is secure and you have a plan to downsize, realistically, the emergency fund you need will be small so I wouldn't bother keeping thousands in savings at the expense of travelling while I still can or improving my home/having in good repair for when the time comes to sell.
The other consideration is whether your retirement is 100% and final. Will doing a bit of casual part time work in your field be an option?