He's presuming. Massively. He's unilaterally appointed you as his carer, but he blew that chance months and years ago.
Obviously you want to facilitate contact for the children's sake, but not on your home or under your supervision. Letting himself in and announcing the new deal without consulting you was a bad move, but clearly all of a piece with his previous behaviour that drove you to separate from him.
He needs to know he can't walk in and out of your home (change the locks in case he had a copy of the key made!) and that you will make contact as easy for him as you practically can but not host it in your home or supervise in his.
He needs to sort his own care as if you weren't in the picture, which of course you aren't, are you?
This could go on for a year or more, and you risk a situation where, the moment you do him any favour unconnected with the children - picking up a prescription, tidying his flat, doing his laundry, etc - it will be your job for the rest of his time.
If anyone fares criticise you, suggest they contact him and offer to do themselves whatever you've said no to. Watch the criticism die on their lips.