Delurking to say: you're amazing! And you're doing brilliantly, but he is still manipulating you. You told him he could collect his things AT A MUTUALLY AGREED TIME and ONLY WITH HIS BROTHER PRESENT.
He led you to believe he wasn't coming last night, waited until 10.30 pm - NOT THE AGREED TIME - when he perhaps thought your friend would have left, and then CAME WITHOUT HIS BROTHER.
You told him he was coming to collect his stuff and NOT to talk or get into any discussion with your friend, but he DID talk and he did get into a discussion with your friend. And he managed to leave without collecting all of his stuff - went upstairs to get things and came back down without them - so he has an excuse to come back again.
You should put it in writing that he broke all the rules you had laid down and he had agreed to, either in an email to him or, preferably, to your solicitor.
You seem to be a lovely person and you still don't seem to want to be rude to him or cause offence. Please, TheName, stop being so considerate of him. You specified rules - mutually agreed time, his brother there, no talking - and he flouted EVERY SINGLE ONE OF THEM. If you give him an inch, he will take a mile, every time.
So when he says his brother will look after the kids on Saturday so the two of you can talk, you don't need to say anything more than "No, that doesn't work for me." You don't have to justify yourself further. Don't say "I'm going to my parents" or "I'm going out" or anything else. As the saying goes (especially on Mumsnet,) No is a complete sentence.
When he says he wants to come and collect some more of his stuff (which he will) then you should say, he didn't stick to the agreement last time, so next time ONLY HIS BROTHER should come, at a time OF YOUR CHOOSING NOT HIS, and he will not be allowed in, because he overstepped the boundaries last time. If he insists = police.
If he really cared about you and wanted to make this work, he would be very careful to respect your boundaries and do everything you want, in order to get back into your good books. Instead, he is insisting he is still in control by coming without his brother, at an inconvenient time for you & your friend, and totally overstepping the mark.
You have been so strong already, please see this behaviour for what it is - yet another attempt to control you and do everything on his terms, not in a way that suits you.