The thing is, this has been going on for years and all that time you have desperately WANTED it to be nothing, normal, just your imagination/overreaction, just something you "made him do" and therefore can avoid next time by not "provoking" him.
Because that would mean you are OK, you are safe, you are happily married, your family is intact.. everything most people want.
And all that time, HE has been pushing the same thoughts at you too - he wants you to believe all that, for all the same reasons as you did, but also so he can still consider himself a "good guy", normal, innocent and a fine upstanding member of the community. Not to mention avoiding ending up in jail.
So I guess you have both been living in a sort of cognitive dissonance - blocking out any facts that don't fit in with what you both wanted to believe.
Now you are starting to challenge this, but that means shaking up your whole view of the relationship, your family life and everything that's happened since you first met. No surprise then that it feels so scary and confusing. But I think once you start to get through this, you should feel a lot clearer in your head, as you will no longer be trying to force yourself to believe things that don't really fit in with the facts.
He, on the other hand, desperately needs to keep the "fantasy reality" alive where he is the good guy (and not at risk of ending up in jail). A part of that is likely to be transferring all the blame for everything, ever, onto you... it was all "your fault", you "made him do it", if you had only done everything he asked ever, he wouldn't have "had" to hurt you....which is bullshit obviously, but a big part of why he is so dangerous to you now. So please, as others have said, do not be alone with him OR allow the children to be alone with him at any point!