There will be other people who know the legal here, but he has moved out, he no longer lives in your home, you must have some rights about whether he can just barge back in as if he still lived there.
He is not respecting you, he is not observing any boundaries, he is trampling over you. It suits him to barge through your boundaries... it makes you feel shit, and he wants to punish you (for something he did, obviously). You have identified correctly that you need to take back the power and control. It is easier said than done, but start by telling him what he can do...don't ask or 'suggest'. Don't ask him to leave 'nowish' - tell him he has to go NOW.
Don't let him ruin your time with the children. When he is gone you can email him a calendar, with a note that if he turns up early you won't let him collect the children until the agreed time. You both need to know where you stand and dates and times are key to this. He has to sign up to a timetable and it has to be in black and white.
I had this for a couple of years until a good friend did me a simple colour-coded spreadsheet showing exactly who had what time at weekends, it also showed school holidays and specified handovers.
The first thing I thought when I read your post is 'Don't let him in the house.' He has moved out. He doesn't get to do childcare in your house. I know it is hard, and that you think it will be hard on the children - but he cannot be allowed to torture you in your own house. You know it is extremely wrong, you in the kitchen upset, while he lords it playing with the dc in the next room. Wrong, wrong, wrong. He doesn't get to do that.
How old are the dcs?
It is his responsibility to have somewhere to take the kids. He says he doesn't want money but fails to find a place that is suitable for them at weekends. Can't have it both ways.
Everyone - particularly you - needs everything clearly laid out, months in advance. If you or he want to change an arrangement, then they need to give a set period of notice i.e. at least two weeks.
Does he come in and do before and after school in your home. Sounds hellish. Basically, you are trying to get divorced from this man but you are having handoevers twice a day? No wonder you're stressed. Other people may have done this may have some suggestions about how this goes on.
The one thing you need to hold on to is that at least he has moved out. Work on that. It may be that in the future you organise paid-for childcare. You do need to get him out of your face.
And what he is accusing you is beyond farce, by the way. His behaviour, as if he is the wronged party is outrageous hypocrisy. For him to try emotional blackmail is almost laughable.
Try and get him out of your face and your head as much as possible, but that's going to be difficult unless you get him out of your home. That should be YOUR safe place.