You've said he can be spiteful, so think about what the worst human beings do when they get their hands on other people's paperwork.
We all take precautions against strangers getting their hands on account details etc, so apply that to him, keep your bank/credit card statements etc safe.
At the very least, he can make your life difficult and cause you expense if, say, your passports went missing and you had to replace them, or he changes utilities without you knowing and there's unpaid balances on the old accounts in your name that are pursued via legal routes. Taking out loans in joint names using utility bills etc as ID is a risk.
Retain pension docs, wills, vehicle ownership docs, etc.
He could just destroy stuff to spite you, precious things that you don't want to lose, exam certificates, birth certificates, photos. Eg, my ex took a load of my photo albums and precious books I'd inherited from my parents, they can't be replaced.
If you have retained any evidence that the £80K deposit on the house came from your money, and that you never intended to gift him half of that £80K, then there's a chance you can claw back more than 50% from the property, even if it is held as a joint tenancy (it's a very outside chance without a deed of trust, you need to get specific legal advice about this).