Your husband behaved appallingly and is a complete knob.
After your first child was born
I was completely exhausted with very little help from my husband.. He wasn't supposed to "help", he was supposed to share the parenting of his child!
I was gradually getting worse and they were considering moving me to Intensive Care.At about midnight, he left. Most loving partners/husbands would have been beside themselves, and scared to death they were going to lose you. And he buggered off home? 
He was uncomfortable on the chair and “really tired” he also smokes and I could tell he needed a cigarette as he was irritable and snappy.
Most partners would have been too worried to be tired or to notice being uncomfortable and would have just nipped out quickly for a smoke. All he thought about was his own creature comforts? He doesn't love you.
I have never been able to forgive him. That is hardly surprising. What he did was unforgivable. You found out in your hour of desperate need that he wasn’t the man you thought he was. Disgusting man. I wouldn't have left a dog in that situation, never mind a human being.
My gut tells me my life will be hard if I stay with him.
Listen to your gut feeling. It is right. He is a waste of space.
It’s so silly that something like this can impact my view on him moving forward in Such a massive way, is it a ridiculous reason for leaving him!
It is not silly and is not a ridiculous reason for leaving him. He does not deserve you and you deserve better than him.
But my fear is being 70+ having health issues and feeling incredibly alone. Regretting a decision I didn’t make 40 years ago.
Don’t let this happen. Dump him and find someone who will love you, protect you and always be there for you.
Our marriage has suffered and my anxiety is awful as I’m petrified of Illness, dying alone or leaving kids behind as he’s always so busy working.
That's hardly surprising either. The first time there was a crisis he failed to rise to the occasion, so why would you believe he'd behave any differently if there was another crisis? You already know you can't rely on him.
He is the type to work when I’m poorly, let me muddle through with kids, if they are ill he will fetch things to help but it’s very much down to me to deal with.
Again this is not how loving husbands and fathers behave. They take time off work when their wives are ill and look after their children. When children are ill they share the parenting of a sick child just as much as the parenting of a healthy one.
He said that he was scared and found it all too much. But his words and behaviour did not show this whatsoever and I know this is a lie, when I didn’t forgive him he then told me the doctor told him to go home. The same doctor that came to me and asked me why he left 
He's also a lying bastard. You can't trust him.
It is not surprising you can't let this go and that you suffer long term anxiety.
I think you should go for therapy again OP and this time tell them everything, starting with that terrible night your so called "D"H abandoned you, when you needed him most.
Then make plans to leave.
