I think the mistake you're making autumn is presuming that an affair is not already underway. I don't think he's wrestling with the fact that he might have an affair and is trying to sabotage your relationship to allow it. I think he's at the stage where the stress he's under from having an affair, coupled with the need to vilify you to provide a justification for it in his head, is producing this behaviour.
I understand your disconnect, but plenty of women will tell you that with hindsight, the biggest clue was that their previously warm, kind, funny and loving husbands turned into a cold, contemptuous stranger while the affair was going on. You can't connect the old H with the sort of man who would have an affair, but as you will see many MNetters saying - there are no 'sorts'. Although having an affair is an active choice and never 'just happens', affairs are far more 'circumstantial' than many people believe. People are often more vulnerable to an 'escapist affair' at pressure points in their lives. You're vulnerable yourself right now in fact. You might not think of yourself as the 'sort' of woman who would have an affair, but stressed people having a hard time with life are always more vulnerable than they (and their loved ones) think. I therefore think you need to see him and what's happening more realistically.
Some of what's happening here though is quite sinister. He is deliberately doing things he knows will upset you and he is also criticising you in front of the children. When the children have talked to you about these days out, have they mentioned 'bumping into' anyone? Have they said what rationale he has given for you not coming?
Also, I don't want to be alarmist but are you counting your sleeping tablets carefully?
You didn't answer those questions about his non-verbal communication with you, but I'd still be interested in your response. I understand why you don't want to hold hands and that's exactly the reaction in you that he intended, of course.
Has he got a work phone as well as a private mobile?
I'm assuming that you're not doing any checking yourself? Again I think that's a mistake. This is a crisis that calls for crisis interventions and you should feel no guilt at all about trying to rule out the most likely cause of his behaviour. Knowledge is power.
Now that you've announced that you intend going home with the children in the summer, you need to lock their passports away in a safe place. I agree with the poster who said that this sounds like you're living with an enemy, but I understand totally why your memories of when he was different are a barrier to you catching up with the mindset he's in now. But that inability to see this more clearly is going to wrongfoot you and impair your decisions.