Amber,
I won't call you deluded but you are certainly in denial and emotional pain yourself.
No court in the land would give an kind of custody to a drunk; those are abusive threats made by him designed to keep you in line.
He is a drunkard; you had an affair probably as a direct result. Two wrongs don't make a right however, and your main priority should now be your children. I am not suggesting at all they are not your number 1 priority but most of your last post again is about him, your H. Many women end up enabling their alcoholic man and that is what you are doing now. How many times have you made excuses for him, covered up for him?.
Your children will love a drunk parent just as you loved your parents when growing up. But look where its got you. You are running the risk of replicating with them exactly what you saw in your own childhood.
You yourself grew up in a household with "functioning" alcoholic parents. It is not surprising at all to me therefore that you married a drunkard as a result; you were taught damaging lessons by these people (children of alcoholic parents often grow up themselves to be super responsible as adults which is not a good thing at all, they can also be left with a wide range of emotional issues) and it has screwed you up despite your denials to the contrary. You growing up in such a house made you more likely to marry an alcoholic as an adult.
Bet you tried to help your parents as well and did not succeed with them; what makes you think you can help your H?. How can you even begin to fix this, short answer is you cannot. Am not being unkind here but as his wife you're the last person who can help him. Besides which he does not want yours or anyone else's help (he has refused counselling). So you end up going around in circles again.
You are now replicating what you saw in your own childhood in these children with them having a drunk dad now. These children will be affected as you have by their Dad's alcoholism as adults and could themselves go onto choose alcoholic partners. This is not a legacy you want to leave them is it?.
Your had an affair which has now left you in a worse position that before. He unfairly now throws all that back at you and you both go around in circles with all this unhappiness both spoken and unspoken that your children pick up on every day. Your H is neither a good man or a good father to these children. If he was a good man you would not have had the affair. I reckon you would not call your parents "good" and nuturing parents either if push came to shove.
You are NOT repeat NOT responsible for him.
You cannot fix him and you should not even try to. Unless he wants to sort himself out there is nothing you can do to make him do this. Coercion on your part to make him do counselling (he has stated he does not want this so what does that tell you?) is an approach doomed to failure.
You and your children are being dragged down by this man. Better to be apart and happier than to be together and miserable.
What are you both teaching your children about relationships here?. You need to think long and hard about that.
How do you yourself see the next 3-5, even 10 years if you stayed with this man?. If a friend was telling you all this what would your response be to her?.
Would urge you to attend an Al-anon meeting, you will meet all sorts of people there with experiences you could well relate to. They are ordinary people from all walks of like who talk about what they have seen and experienced. You need real life support as well. I would also urge you to read Al-anon's literature as it is very informative.
"Codependent No More" is also a good book for you to read.