Has your partner ever been violent OP? That you understandably would have told your therapist about?
I've known a family with SS involved and they're still together, bizarrely with a 6 month old now, but the children are so damaged. Im not sure if it's some kind of postcode lottery, but they've had multiple police calls to drunken arguments, then involvement for about two years. (I posted a thread before under a different name, and actually was the one to alert the authorities, but it was seriously the hardest, but probably most important thing I've done in my life) Still together, fighting, and utterly miserable. I
Your therapist absolutely did the right thing. He's thinking.of you, and your children's wellbeing. Please don't ever stop confiding.
Unless there is a massive backstory, your children won't be taken away from what you've said, but they will be severely unhappy. And so will you. So easy to say but you deserve to live in peace and home should be your "nice" place. You don't want to be walking on eggshells. I think you're a lot stronger than you think! What would happen if you said the same to him?
I don't think everyone should discount @Whatwillnye is saying. My neighbour pays privately for therapy, and admitted she drinks a lot every day and takes drugs, nothing ever happened 