OSJ - don't lose sight of the fact that you never did anything and don't let your dh forget that either. If I were in your position, I would say that I would move out for a week - can you go to your parents? - to give him some space. Like that, you can move back in instead of being stuck waiting for him to move back in. It sounds to me from your postings like you fundamentally want your relationship to work. We all do stupid things from time to time and forgiveness is a necessary part of a relationship.
Can you get someone to babysit and go out to a public place to discuss things tonight? I find being in public tends to ensure that things don't get too heated. If you want to make your relationship work, you are clearly going to have to throw yourself on your sword, metaphorically. But I would certainly point out to him that lots of people have fantasy lives - men watch porn, for example, but that doesn't mean that they're going to go sleep with someone else - and that he is basically contemplating walking away from your marriage because of something which didn't happen and that to walk away for that reason is crazy. Obviously, you cannot overlook the fact that the intimate side of your marriage has not been good for a number of years, but that can be worked on. If he still says he wants to go, suggest that you fix a date to review things in a month or two. That might give you time to get counselling to help you get over your birth trauma. You can still go to Relate whether you're co-habiting or not.
Finally, remember that a break does not mean it's all over.