I really feel for you QueenB, and I know where you are coming from.
I will finally be divorced from my wh next month and I can't wait. However, being divorced does not change things with my child access. Because we were married, my solicitor tells me that we both have Parental Responsiblity, which he wouldn't have if we were single.
My "baby", who is now 6 used to go to his dad's every Saturday for 4 hours (since he was about 2yo). Since a year ago, he began seeing his dad on a Wed eve for 2 hours, and on a Sat for 5 hours. Although I need the break from my ds (ADHD) to catch up with myself, I do worry when he is with him and his girlfriend.
This arrangement has now been terminated by him as he has a new job and shifts. If your wh is anything like mine, he will hopefully tire of seeing your dd, (sorry that sounds harsh) and drop the contact altogether with a bit of luck. It sounds like he could be doing this under duress from his parents. I understand you have been though a rough time with him and he has been a right bastard by the sound of it, but thank god he isn't in your lives full time.
Is it possible to move away from the area, making access a bit more difficult for him? It may halve the time he can spend with your dd to maybe once a fortnight? I agree that you need to see a solicitor about this, and it can be costly - but I am sure it will be worth it in the long run.
I know I sound like a bitch to others who have not been through this sort of thing and believe that a child should have both parents in their lives. However, my ds has consistently NOT been a good parent, and in my and everyone I knows opinion, he could do my ds more damage than good in the long term.
I don't know if that is any help to you QueenB, I really just wanted to say that I understand your problem, but I would definitely suggest you see a solicitor and try and hammer out the access arrangements as soon as possible.
By the way, does he contribute any money to your dd's upkeep? If not, he should be. The CSA may need to be informed, however other MNs may have a better suggestion. PS - Welcome to MN! xx