I don't normally jump in on these threads because I'm in such early days with contact arrangments myself (ex left 3 months ago) but it always surprises me how much some posters seem to think a RP should put up with and 'be flexible' about for the sake of their children having a relationship with NRP. I don't think you sound unreasonable or nasty, or like you expect too much. You clearly aren't obstructing a relationship between the two of them, and in fact would like him to see her more, and on his own. I don't get why you'd be criticised for wanting a routine for your DD and your benefit? 
The legal position (as my solicitor has told me) is that any child has a right to a positive relationship with their NRP. If you do not obstruct that as RP, you are doing your job. It's not your job though to be the one who creates that relationship. Your home should not have an open door policy for your ex, and it is the NRP's job to commit to regular, meaningful contact all by themselves which you are then obliged to facilitate.
If that means every other weekend, fine. If it means 3x tea-time visits every week, fine. It's up to you both to agree a schedule, and for him to follow through on those commitments surely? If he can't be arsed to show up one week and then expects to be able to walk into your home the next, he is being disrespectful and hurtful towards his child and to you. You are not meant to just put up with that in the name of being 'flexible'.
If I were you I would agree a schedule (such as one day at the weekend and a mid week visit, or every other weekend as a previous poster said) and expect that to be what happens whenever he is in the country. IF he's out of the country on work, he just needs to give you as much notice as he has - ie, a fortnight in advance - that he won't be there. When he gets back, the schedule picks up where it was left off, not when he feels like it.
Don't doubt yourself OP - You are not responsible for his relationship with his DD, he is. Don't let him manipulate you any more. But be aware that a court's first recommendation will be mediation, not issuing an order, so I'd explore that as an option. The courts take a pretty dim view of parents who refuse it (except in the case of domestic violence).