Backstory: Growing up my DFather was emotionally and physically abusive - would blackmail me into never getting even a trim for my hair, would hit me for slightest thing like forgetting to text my DM on her way home from work because we'd run out of milk. I was also sexually abuse by my DFathers brother when I was 4, at the time I didn't know what had happened being only 4 so didn't tell anyone until I was a teenager. My Parents split when I was 14 because my DFather didn't believe me and backed his brother, my DM believed me though. Contact was ordered through courts for my siblings, but not me as the court felt I needed to rebuild my relationship with my DFather, they asked him with help to send me regular emails, which was then supposed to build up to short visits with the view that by the time I was 16 I'd be on a regular contact agreement like my brothers.
Unfortunately after 2 emails my DFather thought the process was going too slowly and told me that I either joined my brothers for regular contact or he wouldn't see any of us. I remember clearly telling my DM that I wanted to stick to the way things where, so my DM challenged my DFather about it and he cut contact with all 3 of us.
So that was 10 years ago, I'm now 25, I am married and have my own child, who's 2. Neither of my brothers have children yet.
DBro2 (now 21) contacted my DFather about a year ago. DBro2 tells me that DFathers brother is now in prison for sexually assaulting his Stepdaughter so my DFather now chooses to believe me. At the time I still didn't want contact with DFather but told DBro2 that I was happy for him to be invited to his 21st birthday party in May.
Well obviously my DH and DD are at the party. DFather realises that DD is his granddaughter and wants contact with both me and DD oddly enough he wasn't bothered about meeting DH.
I agree to add him on Facebook and change my settings so he can only see certain posts. And we start messaging. Within a day of messaging it turns to "When can we meet up?" "When can I properly meet my granddaughter?" and offering to buy presents for us both.
I agree to one meeting just before DDs 2nd birthday in June which is nearly a month after we start messaging. I tell him if he wants to get her a present he can but don't expect her to run up and hug him gratefully as she doesn't know him. And as we live in a small flat keep it to something small that she's not going to bothered if I relocate it to my DMs or MILs. He was very good and chose a cushion for DDs bed/chair with her favourite TV character on, she uses it as her "naptime" pillow and uses it on the sofa at nighttime when she's at home.
I've seen my dad once more since then on my own without DD. It wasn't always through choice DD is disabled and has at least a couple of appointments a month, attends a private Nursery 2-3 days a week depending on appointments and also my work. Yes I do also work, but often from home so I can be around if DD needs me. I just got busy and haven't had a chance to see my DM or MIL let alone a man I'm wary of and want to take things slow with.
So he's messaged me this yesterday "No, have you forgotten me? Or have I upset you in some way? I just never get to see you or (DDs name), I miss you both" how can you miss someone you've met once? 
So I messaged back saying that I'd been busy as DDs appointments come in waves every couple of months and she has 12 different referrals in to different departments so I'm fairly busy and that my husband also has an on going medical condition that although not serious requires a little bit of skill to organise his appointments around DDs (he organises his own but likes me with him and as I can't be in two places at once it does need my help to sort out). I also work as does DH and I like DD to attend Nursery on the days I've paid for as much as possible.
So he messaged back this morning "Sorry I'm not important enough to make time for". I want to tell him that I haven't seen much of DM or DBros recently because of DDs conditions and leading busy lives. I also want contact to slow down and be at my pace, I'm happy for him to see DD if/when she's available when I am but he needs to fit in with my busy life and things need to calm down a bit. He never messages me unless it's too meet up, he puts the odd comment on my photos but sometimes it seems he'll only comment if DM or MIL do he'll hav guessed who MIL is based on my surname being the same as DHs/MILs and also because DH has her listed as his "mother" on his Facebook in a way of trying to show he's the better grandparent.
AIBU to want things to slow down? and to work for me? Or should I just give DFather what he wants? I do want to have some sort of contact with DFather but do not want it to be on just his terms.