OP (if you're still reading this), I mean this kindly but you really do need to see the situation through the eyes of SS.
A 4 year old child is pushed so hard by her father that she requires hospitalisation. The mother of that child, although she loves her DD very much & cares for her well in other ways, will not call the Police when this "father" turns up at their home in a violent rage, trying to smash windows etc. This obviously leaves the child living in extreme danger of further injury and, let's face it, witnessing some terribly scary scenes.
If you read that story in a magazine OP, what would you say? This would appear to be the true reason your DD has been placed in to foster care.
Now, you need to have a meeting with your SW and find out exactly what you need to do to stand a chance of your DD being returned to you. Then you need to follow that, to the letter. Jump through every hoop they give you. No excuses. I have no personal experience of Woman's Aid, but it does sound as though they could really help you with this.
If you do nothing, your DD will remain where she is and you will remain in danger from your violent ex. Don't you both deserve so much more than that?