I agree with tinks, I used to drive an hour so that DD's dad could see her and wait up to an hour with her in a cafe when he was late. I didn't know he had a whole other family. He didn't pay maintenance for five years and now pays the total of £30 a month. In short, I went for nine years doing the pick ups and drop offs because it ensured that dd saw her dad. When she was a baby, he would have walked away if I had not taken DD across to where he stayed, for what was often an hour or two of contact. Now she goes for EOW weekends and knows her two half-sisters. Believe me, there have been times when I wondered why I did it, I am certainly not taking the moral high ground here.
My xH2 could never understand why I did this, as he said it was xh1's responsibility. I think that is true, but equally, none of it was DD's fault.
With xH2, there are reasons for my own emotional safety and that of DD, I don't want him in the house.
I don't think I ever felt bullied with xH1, though, he was just irresponsible, though over time, I have come to think it was a bit more calculated. With xH2, I have felt bullied over issues, and I have actually taken a clear line of, this is what I am prepared to do. In your case, I would not drop dc at his door, but offer a neutral location convenient to him. This means that he needs to make some effort, but it also keeps you right as regards maintaining his contact with dc (and yes, I know this should be his responsibility, but no-one is going to benefit from a stand off).