Three (maybe four) years on from a similar situation here.
I'm not going to go into too much detail. I did post at the time, without changing my name, so it is all on here if you search.
Communication is the key. If you feel stressed and worried about things, you must not push it aside and tell your DH.
My DH was very like yours. Mr Matter-of-fact, just get on with it. Part of the reason he was able to allow himself to have an affair was because he was so adept at squashing things into a tiny box and keeping them separate. He had totally separated his work life from his home life, had become an expert at compartmentalising things. So it was 'easy' to have an affair with a colleague.
The aftermath of the affair changed him. No, I didn't kick him out (for more than a day or two), I was 'weak' by some of the standards applied by other posters at the time.
He had individual counselling, which I guess gave him the tools to open up about things he'd usually have kept locked inside.
After the initial shock wore off, he still allowed me to rant, rave, cry, scream, shout and hate him. He never blamed me for the affair, never made me feel I shouldn't be allowed to bring it up, never made me feel that it was time I got on with it.
It's not an easy path. It does take time. I think it took around two years for me to completely let go of it. But now we are happy. And the trust has returned. He had to work hard for it. I'm glad he did.