Definitely have a lock installed and cameras OP. It's always a red flag when a man states his ex is 'toxic'. Yes, she could very well be, as there are toxic women out there, the same as there are men. When you start adding in the fact his daughter has spoken to him for many years and he has 'no idea why', then that's a massive red flag.
Personally, given his lack of boundaries, and the other red flags of him wanting to order food for you etc, I would be contacting the police for a Clare's Law request. The reason I say this, is because my friend dated a man who's ex was 'toxic' and she'd 'stopped him from seeing his daughter'. When the Clare's Law information came back, he wasn't allowed contact with his daughter or ex, as there is a restraining order in place, due to controlling behaviour and violence.
Putting in the request will give you any information that may be relevant. If he is controlling and there's been intervention with the police before, you can then seek help to extract yourself from the relationship.
Sadly, men who push boundaries, use control etc don't generally just accept a relationship has ended. They can become nasty, aggressive and start stalking type behaviour. My friend's ex, who I mentioned above, lent her a phone, when hers mysteriously broke, he'd modified it, so every key stroke she made, he got sent a copy. He had lots of transcripts of messages she'd sent to friends (myself included) and family.
You need to end the relationship. Even without the red flags, he's too old for you.