Start emailing him, no more texts. You need to be able to keep a record.
Keep or get all documents together. Do not give him any docs no matter his reason for needing them.
Document the money put into previous houses and the half deposit and bills paid for current renovation project house.
It will all show on your bank statements so contact the bank for copies and go through them.
Do not agree to 3 out of 4 weeks, go for the common standard of a few hours midweek and every other weekend. You are the primary caregiver and the courts know what standard access looks like.
Get an interest in the house registered. So that he cannot just sell it and then not give you your share. As once that money disappears it's a lot harder to get what you are owed.
Have you got somewhere sorted to live in, or are you still with family. You can use this as a point if you need to take him to court to get your money back.
Stop agreeing with everything he says. Only communicate past pleasantries on drop off/ collection via email. All conversations via email only. That way he cannot twist your words or confuse you. He sounds like a manipulative emotional abuser.
As above do not have discuss things over the phone, email only. Pleasantries and yes access is this weekend at 10am type conversation to confirm what you have agreed via email. But no telephone conversations.
You don't need to be obstructive or aggressive but do not treat him like you are in this together. You are not, he will not allow you to be in this together as he needs to control you. Read up about parallel parenting.
This article explains more
www.scarymommy.com/cant-co-parenting-with-a-toxic-ex/
Good luck op.