Thank you so much for the replies.
I DID leave him. But....I have been plagued with guilt ever since, because the children seem to have emotionally suffered for it. They are older now (20 & 21), but I feel like I ruined their childhood, and I could have just sucked it up, so that they didn't suffer. One of my children took it so badly, and I'm not sure that she's even over it now (despite being 20).
It has been made even harder, because when I left him, his whole family turned their backs on me, despite them knowing what happened, they said that "he just had a bit of fun, whereas I broke up a family".
Sigh.