he says he felt pushed out and I didn't have any time for him and a colleague at work would always have time to listen to his problems and that's how it started
So its your fault then hey? You just had a baby, you were dealing with all of that and poor little boyfriend decides he's not getting enough attention?
Leaving his job doesn't stop the same from happening again. For you constantly worrying that you "need" to make him happy.
he knows what he has to lose now
Can you explain to me how this works? All this time you've been together he hasn't known what he has? Hasn't realised he doesn't want to lose you? You KNOW this is a bullshit thing to say. You had a child together, he just didn't care if he lost you. He was more interested in sleeping with someone else than being home with his family
to feel guilt every time I look at her in the future if I don't just try for her but I am petrified he will hurt me again.
Can you explain how staying with a man who cheated on you, not once, but twice, is "trying for her"? Do you think teaching your daughter that staying with someone who cheats on her is a good idea? Do you think she'll enjoy growing up in a family where her mum is constantly doing things to please her dad because she's afraid of being alone?
No. You teach your daughter that she deserves to be treated with love and respect. She deserves not to be made to feel like it's HER fault when he does something wrong (which you know he most likely will.. such as not arriving on time for a pick-up or having better things to do than take her to the park).
Outside of talking about how you're at fault if you don't stay together, you haven't mentioned what kind of father he's been to her and whether he's addressed that HE is breaking up the family.