I really feel for you.
I discovered on the day i got out of hospital, when my DD was 3 days old, that my 'DP' had been inappropriately emailing an ex girlfriend of his. (I now suspect that an affair was going on, although that email was all he admitted to...he is with that woman now though...) I discovered this as he accidentally forwarded the email onto me.
I never ever forgave him, although i tried to move on from it. I ended up leaving him when DD was 6 months old.
I really feel that what he did, and the fact i stayed and put up with it (but was really bitter about it and we argued CONSTANTLY) tarnished my time with my DD. I should have been resting and enjoying lovely snuggles with my newborn, and instead i was stressed and angry and absolutely completely heartbroken. We argued over it constantly and the atmosphere was horrendous.
With hindsight, i should have left immediately, as i knew even then that i would never trust him again, but i was scared of how i'd cope financially and with being a single parent, etc.
I coped fine, btw. And being a single parent, even though we do now struggle financially, is a damn sight better than being in a relationship with someone i didn't trust and no longer had any respect for.
I feel so sad for you, because when you should be bonding as a family and enjoying the newborn stage you're having to go through this shit.
He should be supporting you, and in complete awe of the fact you've just brought his child into the world, and instead he's behaving like a cunt.
You need to do what's right for you and your baby, and if you know that you'll never trust him again, then my advice would be to not prolong the misery (like i did).