First of all, so very sorry you are going through all this. It must be particularly awful when you are just getting over the birth of your daughter and have a new baby, with all the demands that brings.
From what you've said, even after he admitted something was going on, he continued this relationship? You said you "found out" various things - how? Assuming it's not from him, well done. Your instincts are very powerful and they will serve you well.
Showing such sorrow now is not really a true barometer of his feelings, since some of that sorrow is about being caught and not being as clever as he thought he was. It's also probably the first time he has had to face what he has done and his tears at the moment are not for you, but for himself and for the shit life he could end up with.
I'm afraid the signs are not good when someone is unfaithful this early in a marriage and the repeated lying would really trouble me. I think you also know that there is even more to come out than you have established so far and having information come out in a drip feed is far more difficult to cope with.
You are in absolute shock, but there are some things I'm going to suggest you do now so that in the months and years to come, you will know you made a decision based on all the information.
Without him knowing, register his phone for online billing. You will need an account number and an E mail address. This way, you can keep a track of any contact he makes with OW. If you decide not to ask him to leave, buy a SIM card reader that allows you to read up to the last 20 deleted texts.
In your shoes - and if it's possible - ask him to stay elsewhere for now and say you need to think about your marriage. I think that before he does this, give him one last chance to tell you everything. Then let him go. If he's really serious about being sorry, he won't hedge his bets with OW any longer. If you find that contact has resumed though, you know these are just crocodile tears.
Get real-life support and tell people what you're going through. You need so much support at the moment, especially with a new baby.
Does OW have a partner herself? Find the strength to tell him what has been going on. If she resumes her life without consequence, it will eat away at you what ever you decide to do. Just as you had a right to know, so does he.
Expose them for what they have done, including telling people at their workplace. If he means it about changing his job - he will accept this as a necessary consequence.
I am also a bit dubious about him informing her by text that it is over. To be honest, it doesn't sound likely and if that is the way he ended things, it is cowardly and horrible. Please don't misconstrue this as sympathy for her at all, but I'm always wary of men who treat women with contempt. Hard as it may seem at the moment, if he had ended it with her while taking full responsibility for his own part in events, he would emerge with some dignity. Both parties in affairs are to blame and both of them should be horribly ashamed that they saw fit to wreck someone's life when they were at their most vulnerable.
My instincts are telling me that you don't yet know the full story, that it hasn't really ended between them and that doing it this early in a marriage is not a good portent for the future.
You can recover from an affair, but only if contact is ceased and the whole truth is out there.
Keep posting at all costs.