I'm so sorry this has happened to you.
From my own personal experience when you uncover an affair you go one of two ways; the wanting to stay together, the get that fucker out of my life. Like you I wanted to believe that my ex had made an awful mistake and struggled to adapt to parenthood (my dd was 2 months when his affair started)
I gave him the benefit of the doubt that he was sorry and wanted me but he could break it off with her. I found that out myself as I struggled to trust him and my instinct was right.
My lack of trust in him messed me up as he would say I was imagining things. He couldn't say that when I hacked into his phone account and email account!
Anyway it's been nearly a year and in that time I forced myself to go cold turkey with contact and that space and freedom has cleared my mind.
When you start a new relationship the other person has a blank slate and you go in with a certain level of trust, no matter how much space I've had from my husband and what we've both learnt he'll never have that innocence and clean slate.
I know I deserve more and that happiness isn't just happiness as a family unto, but happiness as me as an individual/woman. It's grasping and believing in it where I face hurdles.
I'd really suggest counselling as it worked wonders for me.
Big hugs - it's an awful thing to go through.