I met my DH at work, whilst single (I had been divorced for a year or so) when my now DH let me know he had separated from his wife.
We'd always got on at work really well and I always secretly fancied him so I was rather pleased, although I needed to understand the wife/ex wife situation before I proceeded.
It transpires she left him a few months before, and moved out of their home. She'd decided to try to bully him into selling the house and he'd decided to not fall for it. So despite her leaving and moving out, ending the relationship, she was of course furious that he hadn't 'chased her' and rather, had decided to move on with his life and for a good few months played the 'scorbed ex wife' to all and sundry.
That was until it came out that she had in fact been seeing her boss for some months, and even before the split.
What I'm saying is, relationships are complicated, people can play a part of the jilted partner or they could be genuine. What OP needs to work out is whether this man cheated on his wife to pursue her, because if so, there's no end to the hell the exW is likely to throw her way.
If he'd genuine been split up and was trying to move on, then of course she's being entirely unreasonable and very unfair on the little girl.
I think only OP knows the true situation, but if he did leave his ex W for her then I think she would be wise to take a step back for breathing space, and see what happens, not least so that he can regain the relationship with his daughter. Under no circumstances should your relationship get in the way of that.
We've now been married for 4 years, together for 8 and are very happy. But I was sure of the circumstances at the time were genuine. If there's any chance your partner left his wife and little girl for you, then you've a long hard complicated slog ahead of you OP. Take a breather, let the dust settle, and see what happens of you ask him for a break. Will he wait? Or will he just run back to his exW? That will be the question.