This sounds a lot like my ex. And like troubled said, I allowed contact to stop. Instead of me pressing him and pestering to come and see my DD or arrange a solid contact arrangement. I stopped and we have barely heard from him since. An event in particular did spark my decision to do that which involved safeguarding my DD, but even so, I realised the only reason he had stayed in touch with his child for the 8 years since we had split was purely due to my nagging!
And like the OP said, its exhausting and draining, because during those 8 years I was entirely made out to be the bad guy. Trying to ruin his life etc he was also to busy with the latest gf, or whatever new toy he had got himself, motorbikes etc to commit to seeing her regularly and like OP would then keep it brief because he 'had things to do'. I saw the hurt it caused my DD but I believed she should have contact with her father so I pushed and pressed and nagged him.
DD has now not seen her father for 2 years. And had no contact during that time, no phone calls, no birthday cards etc. I contacted him about 6 months after I gave up on him to see why there had been total radio silence. He said 'its too much hassle because of having to deal with me, if DD wants him she'll come find when she's older'. My heart breaks for my DD but she is taking it in her stride, now 12 and so much happier and settled for not having an inconsistent parent drift in and out of her life.
I would never actively encourage people to cease contact with a parent, but if you have taken all reasonable routes to try and arrange something and have been constantly met my barriers then you need to stop seeing it as your responsibility. It is his, if he can't be bothered then let him drift away, because he will. He will blame you, but your kids won't.