He might not want divorce either. Divorce isn't some magical clean break where you both live in comfortable homes with healthy finances. He might prefer the idea of looking away at his wife's new friends if the alternative is to sit in a one-bedroom flat above a chippy, seeing his children for a couple of hours every other Saturday.
This is exactly why my DH prefers that we stay together, even though we're 5 years into a sex less marriage. We both hate the thought of destabilising and impoverishing DD, and it seems so needless when we both get on so well.
By the sounds of it, I am in a very similar situation to you, OP. I'd also be delighted if DH took a lover - don't see why he should go on for years and years without sexual enjoyment just because we've turned out to be sadly incompatible in that regard. A few years ago, I suggested that we both look outside the marriage for sexual enjoyment, but he really didn't like the idea - said that it wouldn't suit him because sex = love in his book. We left the question open, but I'm 99% sure he's never played away before or since.
As for my part, I have a high sex drive and am really struggling to cope. At the moment I cant afford to leave so am trying to concentrate on building towards a career that would enable me to live independently and support DD. Maybe if I get involved in a satisfying career, sex will seem less important and I'll be able to go on with DH indefinitely. We've been in counselling and talked lots about separating, but neither of us are prepared to pull the trigger yet. Coping without sex is hard for me, but I find that the more day-to-day happiness and meaning I can build into my life (friends, hobbies, volunteering, etc) the easier it is.
Good luck, OP.
I think you've had an unduly hard time on here from other posters. It sounds as though you're trying to be very honest and upfront with your DH.