Is he a good sleeper? Or rather, was he before having children?
At least two of my brothers, and I, are bad sleepers. It takes a long time to get used to someone else in the bed, and the slightest thing can mess it all up. Stress is a major factor- even light stress can cause disruption.
My DP used to have an issue with it, similar to you- he used to think that the relationship was doomed because I couldn't fall asleep easily in bed with him. I mean, the man snores like a rhino, so there was no way I could sleep next to that racket.
We have compromised, but we still don't spend all night, every night together. When you add shifts, snoring, stress, menopause together, it doesn't make for a peaceful night's sleep with your loved one. If you've only had a child-free bed for a week, that isn't much time to get used to a new arrangement really.
It might be that having a succession of small people in the bed has caused him sleep problems. It isn't always that a man is wanking himself to sleep or having an affair.