First of all, try to calm down!
PR is a bundle of rights and responsibilities, inc. the right to be consulted about educational, religious and medical issues. If granted to your ex, it doesn't mean he can take her away from you.
Most of the time the court will grant PR as it's generally best for the child if the parents can work together, eg on issues such as which school your daughter attends.
Looking now at the contact issue, if he's not happy with the current arrangements, first of all you should try mediation, and if that doesn't work out, he can make an application for a defined contact order.
From what you've told me, he doesn't have grounds for residence (custody).
On the money side, you're doing the right thing by applying for a Schedule One order to have housing during her childhood.
In a nutshell, if you don't want him to have PR you could let him make an application to the court however I would advise against refusing, as by doing so you're likely to force his hand into making an application for defined contact which as you know could be pretty expensive in terms of costs.
I'd advise going to mediation, agreeing within the mediation process to a parental responsibility agreement (simple form, no court hearing) and ask the mediators to help you reach agreement re. the contact.
You obviously need a solicitor to make the appln for your Schedule One order (re the house) as it's fairly technical.