Gruffalo there are some things that you need to realise.
You cannot, are not responsible for, and do not have the power to make him a good/bad father.
Your situation is not a normal situation, you didn't just split up because he didn't do the washing up, or he met another woman even, he was abusive, you had to go through refuge, that is a very serious situation to be in.
I am reading the things that he is saying to you and I can see already that he is manipulating you again already big time. He's doing reverse psychology on you in a very big way and it's working.
Why are you taking the world on your shoulders? Please stop and get some perspective.
All you need to worry about is looking after your dd and rebuilding a life for yourself without him. Don't wear yourself out trying to fix everything when it can't be fixed. There are other people and service who can help you with the contact arrangements, it's up to your ex to prove himself worthy of your dd. Uou can't help him with that, you can't save him.
You say that you're desperate to be loved, firstly you have to love yourself. You can learn to do this by building a life for yourself, being proud of going it alone and making a good job of it. There's so much out there for you, you can work, study, take up a hobby you've always wanted to do. Maybe these things are difficult right now with a baby dd but how about getting out there with her, toddler groups, go to your local children's centre, library, soft play, see what's on.
Second, your dd loves you, and needs you very much, and when you've realised there's more to life than this man, and you start to feel proud of what you've acheived, you will meet someone else.