Stop kidding yourself that you are powerless and helpless to stop this "I can't keep away from him" is really "I don't want to keep away from him"
Take a look at your self esteem here, IME affairs happen when someone is feeling shit about themselves and enjoy the boost that the attention of an affair brings them.
Its said a lot on MN that it is the man who has the commitment to the wife therefore the other woman shouldn't be blamed. But I would say in this case, you knew he was married, you knew he had children yet you still chose to have sex with this man. That does make you a bad person I'm afraid.
And IME a married man who is having an affair isn't there for love or romance or uncontrollable lust, hes there because his own needs are his top priority and the OW gets to be a bit player in his glorified wank. None of his feelings in this are about you, whatever he tells you, it's all about how it makes him feel.
If you are a good person, tell your husband, he has every right to know just who he's married to. Start applying for new jobs. Stop thinking you mean anything to this man. Cut all contact. Go to counselling on your own and figure out what is making you so unhappy in your own life that you would go down this route.