Hello, thought I'd check in to see how you are going.
Great guns, it appears!
I rather suspect if he's mucking with your head now - it'll ramp up considerably once you serve him with divorce papers. Just so you know!
Divorce is not about who did what to whom; who is the least moral person; who is closer to God. Divorce is about separating the assets and making sure the children are looked after in a way that serves their best needs. Remember that at all times. They do need a relationship with him. You don't. Keep up the disassociation....
I'll bet my bottom dollar that as soon as you serve the divorce he will allege, from his pad far away from the marital home that you're an alcoholic, emotionally-abusive and utterly unfit parent. I bet he may even come up with the immortal "I'm going for custody".
Don't even bother getting distressed about this.
A judge won't even ask why he chose to 'protect' himself by staying elsewhere from your delinquency, and why given your inherent instability, he chose to protect himself elsewhere sans children.
Given you are the primary carer and he's kept well away for long periods of time during the marriage - this should be enough evidence for maintaining the situation. If he had concerns about the children he's had ample enough time to deal with that when he was best placed to do so (e.g. not in the middle of a divorce). Spouting this nonsense in divorce is most definitely not the most convincing of times. The same rule applies for you. This is about getting out, staying out and thriving out.
For example, my husband alleges that all of the money he has put in my account for the last 15 years (and which he had total control over) was in fact a loan. Funnily enough, he is now asking for the loan to be repaid.
I did not know it was a loan. I thought it was a marriage. Well, who knew that I was so deluded? Thankfully the paperwork (and lack of) helps me to realise it was a marriage and not a loan.
Keep that disassociated head on your shoulders at all times (and trust me, I've probably run up several hundred pounds ranting to my solicitor about this bullshit - and I realise I really didn't need to). He can say, do, think what he likes.
So can you. Don't put a foot wrong - and you can't go wrong. So don't get into arguments, spats, tit for tats. You've got a solicitor - leave them to deal with it from a legal perspective.
Rant away here - we're all with you, too many of us have walked in your shoes. Some of us are lunatics, some of us like wine, most of us are 'mad ex-wives': None of us have much tolerance for your husband, my husband or most of the ex-husbands.
Trust your solicitor when the husband is untrustworthy. It keeps the madness in check 
.