Yes, if you ignore that it's your home, it's not a particularly great strategy to have £1m+ of your assets in a property - which in the current market is no longer growing in value (and in fact shrinking), and which automatically takes your estate into IHT.
Unfortunately, if you live in the SE - and assuming you can't take a mortgage past retirement age - you have little choice.
I guess you could rent instead... but the UK government taxes rental so heavily (by taxing landlords) that rental is just not a sensible long-term option in the UK.