You don't say whether there is a court order in place for arrangements op. If not, then you need to go with your instinct here and you are right, it all sounds a bit rushed. There will always be those on here who say he has a right as a father to do as he wishes within his contact time. This is both right and wrong because the primary focus is the child. He does not have a right to contact, the child has the right to see the father.
I went through all this with my gs. He wanted his then gf to pick him up from school. I knew she would not want to do this and the Judge intervened stating "I don't want any tom, dick or harry collecting him. Either bother to turn up or don't take him".
I think you and the children are being sensible. You are not saying that in time this will come about, just it is a bit soon for the children. You are not using them to point score, just want to be certain. Don't be bullied otherwise and don't doubt yourself.
If he starts threatening court, then the likelihood is he will get defined contact arrangements.
FWIW, the gf in our case, became the wife and he beat her up too. She is long gone. He now has another new gf and it is all hearts and flowers and he has basically dumped my gs. It is hard to cope with for so many reasons and emotions. Keep on doing what you are doing, providing stability and certainty for the children.
Stick to what you think is right here. However, those saying at age 11 a child can decide, this simply wont wash with the courts. My poor gs is still being forced at 15 and boy does he want his freedom.