A residence order is made by the court where there is a dispute between two parents as to who should have residence - custody - of the child. It will include stated times for access by the non-primary carer.
You are the primary carer for your DD, so you would have residence awarded (unless things went very odd indeed)
However, a residence order will only be made by the courts when there is a dispute over residence/access arrangements. Where the two parties can come to an amicable agreement, there will be no need for the courts to get involved.
This agreement can be part of the divorce arrangements - you can state the standard times when your X will have access to DD - once a week or whatever - and also what arrangements will be made when things change - if for example he wants to see her on Christmas day, or you want to take her on holiday - would you let him make up the missed time before and after?
It's worth putting a schedule together for him to see, and phrasing it as reasonably as possible, so that he can see that you're prepared to work with him to let him see his DD (whatever your feelings on the matter).
If however you believe that he presents a real risk to her, you should be taking more advice on this. It is incredibly hard to get a 'no access' order except where abuse is proven (and even then often there will be access mandated by the court) so you want to avoid him taking you to court over residence if you can, as once there's a formal residence order you both lose flexibility.
But I have to tell you that I am not a solicitor - write to yours, and put all of your concerns down for him, so that he can arrange the divorce in the most secure way for you and your DD?