"Don't you care more about the fact that he doesn't want to completely cut contact with her at all?! Whether he actually does so is a separate issue, the deciding factor in you wanting to stay with him or not is surely the fact you know full well he doesn't want to stop talking to her if it was his decision (which it should be)?"
This.
@Cupcakesaregood - this is so incredibly hard for you, but how are you going to break these circles you're going round in? Because you really are just going round in circles (you've been struggling with the same questions for about 3 weeks now) and I think it's making you feel worse than you would feel if you took decisive action.
There are no good explanations for him retaining the OW's number. You know this. You're frantically searching for an explanation that may make this all okay but you know there isn't one. You've been treated very badly and dealt a huge blow - you got married after a 7 year relationship and 1 year in, shortly after discovering an unplanned pregnancy, you also discover that your OH is having an intense EA with another woman. He tells you he has deleted her number and DC1 is born and in due course you become pregnant with DC2. But you weren't happy in the interim and neither was your H. You've been clear about that. You hoped that a second pregnancy would bring you closer together but all it's done is reveal to you that your H has been in contact with OW all along. Like others, I'm sceptical that it has remained an EA all this time, but perhaps it has. Whatever it has been, it has at least in part stopped him committing to his relationship with you properly.
From things you've said earlier in your post, you seem to be very concerned about appearances. And I can see how you might think it would look unimpressive to have your marriage end soon after it had started. Also, you've said you're 32 - based on that, you've been with your H since you were about 21. So of course he's been a massive part of your life and you've never really experienced single life. But those are not reasons to stay with someone who doesn't love you and no-one sensible would think any the less of you if your marriage breaks down.
Your marriage may be salvageable but your H has to feel motivated to commit to you ahead of the OW and I think knowing that you will hold on to the relationship at any cost - as appears to be the case - is making him more careless about this than he might be if he thought he actually stood to lose you.
But given that your marriage has been unhappy for the vast majority of the time you have been together, it may be better in due course (perhaps not now but after DC2 has arrived and you've all had some time to adjust to her/his arrival) to let your marriage go and concentrate on being the best co-parents you can. You so deserve to be with a man who wholeheartedly wants to be with you, and from what you've told us, H may not be that man.