Cherry start keeping a diary. Write down as much as you can remember from him being unreliable too. Hopefully you won't ever need it, but it's better to take proactive steps now should you find you do. NOte dates, times, what happened (eg dd2 went to friends as he found her behaviour challanging).
You do NOT need to go to court to maintain your situation. He will need to take you to court to change the current arrangements.
Perhaps write a non confrontational letter & keep a copy for future should you need it - say in your letter that you hope he will now make more formal arrangements to see both children regularly, and how it's good to see he is now realising how important it is for the dc to spend time with him and their new sibling. Also mention that you appreciate he has been more reliable lately and this is important for the ds that he continues to maintain this.
Basically covering yourself to show you are thinking of the children and pointing out that this is different to how things were before should you need to use it in court.
I wouldn't mention anything at all about dd1 saying she wants to go and live there in the letter. It is a perfectly normal thing for a child of that age to say and she does not understand that it means she won't see you if it happens.
If he takes it to court the CAFCASS officer will look at what is best for all the children, not just one. From what you've said he wouldn't have much chance of getting sole residency of dd1. Let him waste his time and money in court if he wants to try. You don't need to do anything, just keep loving both your dds.