When he complains to you that something you want is 'silly' what he means is that he didn't see that one coming, and is perturbed by your ability to assert yourself, as it is a very unwelcome dose of reality for him. It interrupts his little fantasy.
Stick to your guns about his coming and going to the family home, and please do not budge. He is testing your boundaries. If he shows up unannounced, tell him you will not admit him and remind him that you have made it clear that you require him to have your permission. If he kicks up some sort of fuss, you are within your rights to call the police I know this may seem like overkill, but you never know. Please don't let this slide make him stick to this requirement of yours. It's a very important hint by him as to whether he has any intention of playing by the rules, or showing any signs of respect for you, or whether he thinks he can have everything he wants, his own way. He will teat you, and he will not like it when you resist his pressure, but be assertive (not necessarily angry or aggressive, just state what you want calmly and firmly.)
You really can insist, and there is nothing he can do about it, because it's a very reasonable request. You have not told him he cannot under any circumstances come to his own house, you have just requested advance notice. Perfectly reasonable. And watch his reaction. You already have an inkling that he is pushing back. That's that sense of entitlement of his kicking in, the same sense of entitlement that made him think he could find himself a fuckbuddy (boak), leave the way he did, speak to you so coldly on that dreadful day last week, have a friend deliver an insulting letter through the door out of the blue...
I would be very hesitant to agree to anything he wants wrt visitation until ordered by a court. He has clearly put some thought into how things will be (he has had time, after all, to give it all some thought). You are more likely to end up with some sort of every other weekend visitation schedule, which is standard, and if I were you I would tell him you will be consulting a solicitor and will get back to him, and until then, no solid visitation arrangement will apply. He can just keep on asking you if he wants to come over (and I hope you are also telling him no phone calls at random times to you), and keep in contact with the girls over the phone at certain times, but that they will not put their lives on hold waiting for him to call -- they will go to parties or other activities as arranged. this phoning thing could very well turn out to be a way for him to seriously annoy you and try to get the girls' lives revolving around the time he calls.
Take every opportunity you are presented with to burst his bubble.
And yes, you are right, the idea that the ball is completely in your court now, that you hold the future in your hands typical pathetic effort to shrug off his (100%) responsibility for this. You do control what happens to the extent that you can fight for the best possible life now for you and the girls, but is this upheaval or any future divorce your fault, your responsibility? NO. And he is a coward to suggest this. A cold and calculating coward.
Hoping you had a restful night, and hope the duvet day goes nicely. xxxxx