Firstly what struck me is that you said you pushed previous problems aside when you were diagnosed, not that you dealt with them. I can't help feeling that this would be a contributory factor in the mix of what's happening now.
Secondly, my mum has a rare form of non-hodgkinsons lymphoma. It can't be cured but it can be managed and shouldn't kill her. However, I'm very fearful before every 3 monthly appointment, it doesn't get easier. My mum is the person I'd normally talk to about my fears in life but I couldn't in case it worries her. I talk to my dh and he understands how being fearful feels because he's lost both his parents and neither ending was a good one. Despite the fact that she shouldn't be terminal, he understands my fear, cares that I feel that badly, and doesn't go into competitive who should feel worse mode.
I'm sorry that sounds harsh and I'm aware my own experiences have coloured my impression of your post, but it reads like you're not understanding of the stress your condition placed him under or allowing him to feel fearful. There's only so much of that anyone can take and 3 years+ is a long time.
Of course, I'm very much influenced by my situation, but in a healthy relationship people need to be able to freely express their fears, and the other person can empathise and wants to ease that pain. It's seems that neither of you is open to the woes of the other, you've already said you've pushed yours from pre-diagnosis aside and it appears like he might be doing the same. In addition neither of you are sympathetic or wanting to help one another.
Unless you can find a way to be able to listen to each other, acknowledge the feelings of the other, and want to make the other person feel better, you are only going through the motions and blaming each other for any shortfalls.
The question is, do you both want to?
As I said, I might be completely wise if the mark and if I am, please ignore and don't take it personally as I don't know you personally 