OP - here's another angle on this. You do realise that what you are actually doing by facilitating all this is ensuring that your children end up with a sponging, controlling, aggressive, nasty, bad-influence older adult in their lives, ALL their lives, doing to them what he does to you?
You AREN'T giving them the illusion of a lovely family with a great dad. You are setting them up to be swindled, harrassed, sponged off. The way he is with you is the way he is FULL STOP, with people close to him. He is simply a bad egg - what you've said about the house is unbelievable! He's STOLEN from your DC - what the hell do you think he'd be like with them once they have homes of their own, stability, maybe more money than him? He'll make their lives a misery.
He clearly doesn't give a shit about them. Rock the boat because with any luck it might make him FUCK OFF.
Please go to the CSA and get it changed and tell him you don't care, he's not coming in your house again. If he wants things changing, take it to court. He can't take them to his house? His problem. Oh, and his house is actually THEIR HOUSE. So get to a solicitor and start sorting that too.
You have allowed your children's future security to be stolen from them by a man you're letting primt them now to be a leech for the future.
Start stopping it now!