I know exactly what you mean about feeling like you're bluffing in a grown-up world, Make! I often feel the same about life! 
There is a lot of confusion (on this thread and elsewhere) about what parental responsibility actually means. This leaflet from the Children's Legal Centre may help clarify some things...
There is NO connection between PR, contact and maintenance - all three are separate issues under English law. A PR order gives a parent the right to be involved in major life decisions - major medical treatment, where s/he goes to school etc... It says nothing about contact.
(So for instance, my DS1's dad was granted a PR order at the same time as a contact order was not made, because he told the court he was moving 150 miles away and didn't know if/when he'd ever see DS again. And he's never paid any maintenance. In practice, the PR order that my DS's dad obtained 15 years ago has never had any impact on me... But it could have done - if for instance we had disagreed about his religious upbringing or schooling... But since he was totally disinterested, he never sought to have any influence at all).
There is a presumption that parents with PR will co-operate in the best interests of a child. If they don't or can't, one parent can apply to a court to resolve specific problems, like contact.
Your Ex can apply to a court for a contact order, Make, if he doesn't like the contact you are offering him. This has nothing to do with his PR status - he could do this even if he didn't have PR.
A court will agree that it is in your DD's best interests that she sees her dad. It will probably agree that he should have unsupervised contact, unless you can truthfully argue that she would be at risk from him. It will probably agree that, at her age, little and often is best.
It is unlikely to agree overnight contact, given her age, and the previous lack of contact - but you should expect him to be granted that at some point.
It will not rule that you have to drive to 'deliver' your DD to him. It very definitely will not say that you have to drop everything and agree to contact at 24 hours' notice. You certainly don't!
An important thing to understand is that a contact order requires you to allow DD to have contact with her dad. It lets him have contact; it does not make you do anything at all except allow him and not be obstructive... This leaflet from Gingerbread has more info.
If you have clear, documented evidence that you have offered him regular contact at/near her home, a court is likely to think this is reasonable.
You could also offer him contact via Skype - e.g. he could read her a bedtime story - which would be a way for them to build up a relationship without him having to travel, and you could 'back off' a bit and let him have time alone with him while knowing she was physically safe in your own home.
You could try a kind of 'broken record' technique - send him an email once a week offering specific contact times/places (e.g. one or two alternative 'physical' meetings plus Skype contact) and ignore everything else except any polite request from him asking for an alternative time.
I don't think you need to worry about a Residence Order - but I'm not a lawyer, and if you want to be sure, you should ask one. He has never been resident with her, and you always have been, and courts generally maintain the status quo unless there is a very good reason to change it - and there isn't in your case. His PR does not strengthen any claim to residence he might make - that would be decided based on your DD's needs/best interests and his ability to care for her.
Honestly Make, you don't have anything to worry about. He is angry that he is not getting his own way, and he is trying to bully you - it's just noise.