OP, this is such classic control and abuse that with your first post, I knew, before you actually said it, that the thing you were withholding was that you'd spoke to another person, probably a man.
It's text book. It's not unique. It' snot becuase he has had this specific, unique set of circumstances that make him insecure. It's just classic controlling, manipulative behaviour.
My response would be to ask him, "why are you with me if you don't trust me".
For the record, men like this are very evry good at targeting women who are strong and independent, and emphathetic. You can see his side, so you think maybe there's some reasonableness. But there's not.
It's also not that unusual for this to go on for a very long time before you really start to push back. In part, it's the boiling frog issue - ie that it builds up over time and you don't realise quite how bad it's got until it's too late. But also, partly I think it's becuase often when it starts, you don't feel the same need to push back. Youo're in the early stages of love so happy to spend all your time together. You're a bit insecure yourself. You have young children so not going out with friends isn't that big a deal.... and then slowly, you realise how insidious it is.
Also, it gets worse and worse over time so before maybe it was becuase you spoke to a man in a pub. Now it's becuase you smiled at a man on the school run.
Quick warning, when you push back you are probably going to get one, or more likely both, of the following:
Accusations that you are a terrible person, manipulative, toxic, dishonest. A terrible wife and mother. A bad person etc etc.
Lots of tears about how he can't live without you, that he knows he's a terrible person but he can' thelp it because of his "trauma" and that ONE time you did something mean to him 20 years ago that he's never been able to let go. Boo hoo hooo.
If you actually leave, or ask him to leave, these will ramp up - in the first case to even more accusations, aggressive behaviour (possibly violence), unfounded accusations, likely a smear campaign to friends and family. He'll go out of his way to make your life difficult - take your car when you need it for work, not come home in time to give the kids' dinner, not collect the kids from school on his day etc.
While on the victim side he'll probably claim to be depressed, refuse to work (lose his job), threaten suicide etc.
What do your family think of him,? If he's tried to keep you from them, I bet they see right through him.