This discussion is breaking my heart.
My father left when I was 2, and my brother was 3. It was a very acrimonious divorce - my parents loathed eachother, and made no secret of the fact.
My father scaled down his visits over the years, and eventually stopped visiting when I was about 12. I have no idea what his reasons were, but maybe he found the brief contact with my Mum unbearable. Or maybe he felt he needed to "sort himself out" in order to be any use to us.
Whatever his reason, it was a big mistake. The feelings of rejection we experienced were immense. Although I got on with my life, I never re-established a relationship with my father, despite his attempts to have regular contact now I'm an adult. We e-mail, he visits, but I feel nothing for him. My brother wasn't so lucky. He took it very hard, never got over the fact that someone who created him could so easily just drop him. He committed suicide when he was 19, and his diaries spoke at length about the devastation caused by his father's abandonment.
I think that when you bring a child into the world, you have to put them first, however hard you may find it. I split with my ex when my son was 3 weeks old. I had to move out, despite it being his choice to split. It was very hard - moving house with a tiny baby, leaving the person I was in love with. But that's life - I had a son, so he came first.
Please, as much as you may be suffering, put your children first. And don't justify hurting them by telling yourself you'll be more use to them when you're more sorted. You're their Dad, they love you, and if you drop them now you may lose them for ever. I'll never ever forgive my father for what he did to my brother. Never.