intheend - HUG this is a 'giving you strength' HUG, not a sappy 'there, there, pet' hug. Right, take it, hold onto it and stay strong against this bully.
There are so many good posts here, please read them all again and the links. Realise that what he is saying to you is so completely 'text book' abusive behaviour.
No one else would want you... that's what they all say. So many women believe it, because they have been told it again and again. It is rubbish, it is a way to control you, nothing else.
He said yesterday he wants 'Sweet Revenge' can you imagine what kind of a beating that will be if what he did the first time didn't 'keep you in line'???????
Tell your family - tell them everything, then he has no hold over you. He cannot threaten to tell them about your affair. I am sure that once you tell them what he has been behaving like, your affair is going to be a big, fat, nothing to them. They may love the person they thought he was, but they love you for who you actually are.
You asked what kind of responsibility do you have for his behaviour if he beats someone else up because you wont let him beat you up move back. NONE. His behaviour is his responsibility and his alone. Don't even go there, you have enough to worry about already.
Do not let him in, get a restraining order. Keep this bastard away from you and from your children.
[I haven't been in this situation myself - which means I don't know how you feel, but it's clearer to see what is really going on]
Stay stong!!.