Haven't read the whole thread I'm afraid but I really empathise. I spent a few years in my late 20s in a really similar on-off thing with a guy with a long history of commitment-phobia; I tried to walk away repeatedly and kept failing. Like you we spent a long time "not together" but behaving like a couple in lots of ways. I was so convinced that if he could just work through his issues we could be happy, and that that was what I wanted. The shift for me started when, after a lot of thinking, I reached the conclusion that actually there had to be something in me that was perpetuating the situation as well. Even though I really believed that I wanted him to commit and settle down, and his constant shilly-shallying genuinely caused me a lot of pain, in the end I had to admit that something was keeping me there, in an obviously unsatisfactory relationship. So to some extent and on some level it must be what I wanted. I found this realisation very hard to accept but it was a huge shift for me - as I realised that it was as much about me as it was about him and although I could do nothing to change him, I really could work on myself.
At around this time I moved away for work anyway, which helped with the distancing from him (though we still ended up spending Christmas together!); I was really enjoying my new job and started some therapy to tackle my own feelings about intimacy. I also did some casual dating at this time, but nothing serious. Over a year I made quite a lot of progress in identifying why I might, deep down, be very anxious about intimacy myself and began to feel more ready for a "real" relationship.
After about a year of this I met my now DP. We were friends a bit, and then had a couple of months of 'courtship' and then got together. I was very honest with him from the beginning about my relationship history, and that I was aware that I had some fears about intimacy. I'm glad I did because I really fell for him, and then a couple of months in I hit an absolute WALL of panic, really similar to what you describe your guy as experiencing. I felt unbearably anxious, cried all the time, kept getting ill (v. obviously psychosomatic!), it was awful. I felt panicked by his presence but also furious and upset (like a toddler!) if he so much as left the room! But I was really determined to work through it, I kept talking to him, I started seeing a therapist again (I'd moved again by this time), and (crucially) he was fantastic about it. I won't lie, it was really tough on us both for several months but a year or so on and we are living together and I am expecting a baby. I still have difficulties at certain points - especially if we've been apart for a bit - but we both have a handle on it and I really feel that I have got beyond something that had been silently dictating my emotional life for years and years. We both did a lot of reading on "attachment" which I found very helpful.
After about a year of not seeing my ex at all, to protect my new relationship, I now see him occasionally as a friend without any problems or angst (or longing!). He is good company, but to be honest the extent of his "issues" are now very clear to me and I find it quite sad, as he is in his early 50s now and I don't think he will ever be able to have a satisfactory relationship.
Anyway, I'm not sure if any of that is any help to you but I suppose the key thing is that I was able to begin the process of walking away when I recognised that the situation was as much about me as it was about him but that my own feelings and actions (unlike his) were something I could work on and control.