If it helps, I think it's normal to feel or focus on loss. When I talked to Women's Aid, I mentioned the pain I felt sometimes when I saw happy couples pushing a baby around together and looking in love, or when I sometimes felt that single parent stigma at my daughter's nursery, at collection time. Hearing people chatter about their husbands and their "normal" couple lives.
The thing is, I wasn't mourning my ex - I didn't want to be with him. He let me down very badly. I was mourning what could have been, what I had hoped for. And disappointment for my DD, as I didn't wish this situation on her. But as the WA worker pointed out, it was much better for her not to grow up witnessing his behaviour towards me.
You will have had some "good" times, of course you have! So did I. It's the only reason I didn't leave sooner, because it's part of the cycle of EA. Easier said than done, but try not to let them cloud the present and the overall reality of your situation.
Think about the last time that you really had a properly brilliant time with him, and as a family. The thing is, it shouldn't be hard. There should be happy times all the time. I feel very lucky to be in my new family set up where my partner can't wait to get home from work and play with my DD, and sets aside every weekend that we have her so we can do things together.
I should have left years earlier, just like sassandfaff.