BNBH - you'd know better than us whether he is a twat or not (although you may have a personal bias of things like that but the main thing is - is it in your DD's best interests to spend time with him? Is he a good dad?
It's the hardest thing in the world to seperate your feelings about an ex from those concerning children - but parenting's like that anyway! It's not easy when you're together either.
It's good that you're willing to think and discuss about this sort of thing. From a father's point of view (and one who usually gets once a week' and what's more is deeply unhappy about it having only got that through an ever-lengthening court battle) assuming he is a half-decent bloke, the best thing to do would be to <span class="italic">talk</span> to him. If you can't do it civilly, try mediation with a professional to referee' and keep the discussion on what you're there for - your child.
It's a slippery slope, BNBH. In my personal case my ex left and refused to talk to me. So I organised professional mediation. She refused that. I took her to court. She made accusations of DV and child abuse against me. That's been forgotten, but we're still fighting on. Even last week I wrote a letter offering mediation in an attempt to break the deadlock and that was ignored.
We can all appreciate it is hard - but if you start sliding down that slippery slope it will get harder - you will get entrenched, he will get entrenched and you will both be in holes so deep neither of you will be able to get out. If you can't agree with your ex about things - a stranger - i.e. a judge or the like will make a decision - quite possibly one you, your ex and most importantly your daughter won't be happy with.
Keep talking to him BNBH. You never know - in a year's time you may look back at the stress and apprehension you feel concerning your daughter and your ex has been washed away and you see him as an ally in raising your wonderful DD...rather than an enemy for the rest of both your lives.
Believe me - the alternative ain't nice!
Oh - and get him to join Families Need Fathers - he may need reassurance about his role as a father and they will be able to give that as well as lots of help on how you can both work together.