@Alcemeg gives you some good advice, OP, but many of the replies (as always with infidelity on Mumsnet) are infantile and patronising and I would skip through them.
Would you say to an alcoholic 'just go cold turkey!'? In the long run, yes, it will be the case that you need to stop contact with this man, just as an alcoholic needs to completely give up drinking. But, equally obviously, this isn't easy to do. If it were, alcoholics would all have given up already and people in destructive relationships would all have walked out!
How is the sex? How does it compare to other sex you've had? Is it less vanilla? Is there a faint BDSM tinge? For some of us (and I too have had boyfriends, though not affairs, that fall into this pattern) there is a strangely masochistic pleasure in being in a bad relationship with a person who treats us like shit. I'm not sure that this is because we believe that we're fundamentally worthless, though perhaps that can't be entirely excluded. But there can also be a pleasure in the sheer simplicity of submission, not making decisions, asking 'how high?' etc. It's almost the perversion of a desire that we might have gratified in the past through religion, adoring where you don't always believe, submitting always to someone else's preference. Etc.
Because you know that, basically, this is a shitty and destructive relationship, and because you're ashamed of it (and presumably constantly locked into a stressful battle to conceal things) there's also a huge sunk-cost factor. Leaving the affair means admitting you should never have had it, that it was all a waste of time etc. This is a classic fallacy, of course!
Try to find some tranquil time for yourself. Swimming, a long walk, time alone with your own thoughts. Force yourself to use some of the time you spend daydreaming about your affair partner constructing realistic mental images of future possibilities. Not only what will happen when your husband finds out, then your kids etc, but also what will happen when your affair partner goes no contact suddenly, or breaks up with you, or turns out to be having a second affair with someone else. (In my experience, men who have affairs of this kind are not usually shagging around randomly, but they often have a string of affairs, one after the other.) And it's also very likely that if you end the affair now no one will ever know. It suits people with no sympathy to imagine that all affairs are discovered but this is very far from being the case.