Does one child really need this whilst we could enjoy some of it by not paying down all the mortgage? It would seem that paying off a mortgage is partly storing up money for others to enjoy, including the taxman .Just to be clear, I appreciate the peace of mind argument for paying the it early and less the interest paid.
You need to see an IFA.
Separate out all of your goals.
Now, retirement, and inheritance. They are all separate issues.
I could ask why you need a house that's worth £whatever, if only 3 people live in it. Why not sell it and buy something smaller?
Why do you want a big and expensive house (and mortgage) at the expense of not having holidays? Because that's what you are suggesting - not overpaying your mortgage so you can have the cash now to spend.
There is lots you are not accounting for in your plans. Things that might happen.
Care home fees for a start. You wouldn't have to sell your home upfront, but the costs would be clawed back (as a debt) once your house was sold, unless you had cash to pay the care home fees.
Without knowing your circs, it's impossible to advise, only point out the things you aren't thinking about.
Some people want to pay off their mortgage in case they are made redundant (how secure are your jobs? Do you have savings to cover any shortfall?)
How much income do you expect to have as retirees and would that cover your mortgage?
At what age ideally do you want to retire?