Right, stop torturing yourself - he cheated. You know this. So park it. He's a complete shit - you also know this, not only from what he did but also from how he's treated you since it blew up. Take your daughter away? Wow. Repellent pig.
Here's what will actually happen if you leave and he decides he's going to 'take your daughter.'
You would pack your stuff and go to your mum's with your DD, leaving a satisfyingly FUCK YOU note which makes it 100% clear that the belittling, abusing, and the CHEATING are over, and he can spend his time washing his own pants and thinking of ever more inventive ways to convince Mummy that he's a good little boy really.
You get settled, find yourself a flat/get sorted with your parents, get daughter into preschool, start applying for jobs. All could be kicked off within a day or two of you leaving, all of which embeds you in the place you have moved to, and gets your daughter settled in ways which a very strong case would need to be made to overturn.
He would take at least this long and more to even get a couple of appointments for legal advice. Meanwhile, he can do NOTHING. You contact him to let him know that you are happy to arrange contact through e.g. your mum, but you don't want to see him. You would not have to take daughter to him- you would state in writing that you believe that he might try and prevent her leaving, so you offer that either he comes to you and sees daughter in presence of your family OR he arranges a contact centre himself. All this would be explanable in court, as he has made threats to remove her from your care. He rants and raves - you reply calmly 'Take it to the court.'
A decent time would pass before any court date came up. By then, you and DD would be settled.
He would never in a million years then get the result he would want. Your case would be that after the relationship broke down - and I'm sure he would enjoy the court hearing why - you, as primary carer, had no choice to move to where you had financial and emotional support to care for your DD while being able to continue working. You would by then hopefully have a job, a home, and DD would be either in preschool or have her name down for one.
So, if I were you I would leave without a thought. This man is a pig - a prostitute using, lying, abusive pig. He isn't going to get better, you aren't happy with him, your daughter certainly won't have the life you want her to have. Better a happy separated home than one full of resentment, hatred and growing up seeing one parent bully and disrespect the other.
Leave!