A young child needs contact little and often. Once a week for an hour doesn't cut it (I say that as it seems to be a common contact regime that is rolled out a lot).
In an ideal world, an hour a day would be good. Note I say `ideal'.
Truth be told there is research to both back up and refute what both you and your ex want. A lot of received wisdom too - for example there is a distinct Young children need their mother' known as the Tender Years Doctrine' and enshrined in British law in the Custody of Infants Act 1839.
In the modern age the legal view is that there should no presumptions and cases should be looked at on their individual merit. There is nothing in theory to stop your DD spending time with her dad so long as you have made provision for feeding.
In reality however he is unlikely to get overnight contact at this stage. But when it comes to unsupervised contact'...ask yourself if this is about your daugher's best interests or what YOU want (I say that because you say m not ready yet to be apart from her') - would this be relevant if HE said it?
You have no rights. Neither does your ex. Your DD does though. She has a right to a meaningful relationship with both parents and it's in her best interests to grow up seeing her mum and dad working together in her best interests. She is half you, half your ex. The fact that he has PR means he is identical in law to you. You may be the `main carer' and you may be identified as the first parent anyone will come to by virtue of holding the child benefit book but that's it.
When all is said and done however, he isn't helping his case. Not turning up will be frowned upon by the courts. So will making threats. Failing to return her will see him end up in contact centre if the matter reaches court too.