You can move, you just have to make sure that equivalent levels of contact are made available. As dd is not in school yet there is ery little that he can object to that a court would take seriously and if you have a plan in place that will make hera available for a reasonable level of contact then he would most likely get no-where.
WRT his behaviour, well, going back to court would mean you could explain all of this and explain that you do not feel it is in dds best interest to be exposed to that kind of behviour. The court will, most likely, involve caffcass who will do a report. If things are that bad then they may ask for supported contact for a while, hopefully the process would show ex how importnat it is that he does not behave like this. If it doesn't then you have your case wrapped up.
He is bullying you and using dd, if you hand her over he will use that against you too (Mummy dosen't care, she wanted rid of you) and you will be in no position at all to deny it or put your side across. You really will not be doing any favours to you or dd by going down this route. The chances are that you will lose contact with her altogether in time.
Hard as it is sometimes you do have to let these things play out. Dd will see that the things daddy says are not true - it may take years though sadly. The more he says these things - and the more dd tells people what he is saying the more he hangs himself if you take it back to court (or he does for that matter). Keep reassuring her that you love her and that you would never hurt her do not enter into discussion with her about the things he says unless she has specific questions. Just reassure her that you would never hurt her and you love her every time it comes up. Do not get into "no I did't dd, your dad is just saying that to..." because you will become the same as him in bith dds and a courts eyes.
Get a solicitor, get some advice and stop letting his words affect you (yes I do know how tough that is) his words are just words. I found the best way was to behave as if they were funny "Mummy, dad said..." you say "really? what a silly daddy he is never mind dd, you know I love you don't you?"