I'm going to try and message you. As I think really even if DD's DF isn't Father of the Year material, I think having contact is important, I also think yes you've had issues with other family. It's important not to relay your feelings onto your children, not saying you're doing that, I'm just thinking children ideally need both parents in their life, unless something awful happens, which makes it a no go.
He says his life style isn't conducive to children, I wonder, it could be conjecture, how much of that is you saying, look you're not in the right place to be an active parent, his confidence doesn't allow him to say surely I can have some contact.
Parents who do it by themselves sometimes get a complex where they'll say to DC, I do everything for you, what does your other parent do? It creates this situation in a child's head where they think well why? If life was straight forward that would be amazing, it seldom is.
I've posted on here a few times, rather than come up with a counter argument, maybe sit on what I've said for an hour or too, see if anything resonates at all. If not perfect, if you do feel superior as the sole carer, you need to address that.