Many people in the SE could be considered a millionaire if you add up the house values.
The important factor is an imbalance of assets, however small.
It's your long term future you need to consider and what happens if he dies (young, or old.) What's he doing with his Will? Is this part of your conversation?
You need to plan long term. What happens to his estate? Would you have to find a house of your own again- would you relocate back to your old house (as it' s not in the same area.)
These are big questions.
You may think it's all decades off but people can die young so you need a back up plan.
FWIW I relocated to when I married my DH. I also gave up my job . I was adamant that I'd not live with him as I'd be putting myself in a precarious situation. I'd not have been able to afford to buy where he lived (he already had a house he was buying and his earning power was far greater than mine, despite us both being grads with professional jobs.)
I know you don't want to get married again but it gives some financial protection.
I agree with the PP above ^^ that it's down to you to create an escape plan and not expect him to manage that in any way.
If neither of you don't want to be married, that comes (in your case) with the risks of having to start over again with no share of a joint assets.
You do have a house - which is great- but of course it will be a long way from where you're moving to with him.
If it does turn nasty, the simple facts are you could have to rent an Airbnb long term , or a short term rental of 6 months, till you can get your tenants out and then decide what your next move is.
Neither of you will want to be cooped up together for 3 weeks let alone 3 or 6 months if you decide it's over!