These posts are, by and large, very conservative.
I'm with Lauriefairycakes - I think you should consider an open marriage.
And I can see that if your dh is avoiding confronting the issue, it's going to be really hard to have that conversation, hence considering an affair.
This has gone way beyond the point where getting rid of the kids for a night is going to do any good.
I can see why you don't want to end your marriage - in many ways it functions. Why are posters here so either/or? There are lots of ways to conduct a relationship.
Agree that honesty is best - but, weirdly, some people don't like that. My suspicion is that your dh is one of those. It is, unfortunately, quite possible that what he wants is you to have sex elsewhere and just not bring it up. Be discreet. Some marriages actually function like that.
My other guess is that you, sadly, are not like that, and would prefer honesty.
Whatever - you have one life and you are letting your dh make you v. unhappy. Maybe everything else is good, but this is so major, you must take yourself and your happiness seriously.