I can offer some insight to a vaguely similar situation.
Ex h and I split now about 3.5 years ago after I met someone and it was the catalyst for me to leave, I made some poor choices in terms of that, but it came after 15 years of him cheating, physical and emotional affairs, constant lying, emotional abuse, controlling behaviour, I was only 16 when we met and he was in his mid 20s and I was totally under his spell. I met someone and nothing much particularly happened but it gave me strength to finally stand up and leave, 2 dc's who were 3 and 1 at the time, so ages are different to yours.
After we had been separated about 3 months ex h took my phone and discovered the overlap of meeting someone and me saying I wanted to separate, so he has always been seen at the affair partner to him. We are still together now 3 years on but I have kept him separate in a lot of ways to my life for most of this time for fear of my ex's reaction. Some examples of his behaviour over the last 3 years are stalking me, hidden cameras, stealing phone, listening devices, looking through my windows at night to see what I'm doing and listening to conversations, putting in place huge restrictions on what I can do, delaying divorce proceedings to the point we are still not divorced, lying about money to lawyers, I could go on.
About 6 months ago after he introduced our dc's to a girl he was seeing I decided I couldn't go on keeping mine and my partners life so separate, that although I don't plan on jumping in living together, I've found someone who is a wonderful person and partner and I don't want to live our lives so separate. I had to suck up what my ex's reaction would be because I've lived in fear of it for so long. I spoke to him about it first and that I wanted our dc's to meet my partner, he's never been 100% ok with it and I get snide comments from him all the time, but it's not been as bad as I thought. My children are a lot younger than yours and I do have to say I think that if they were both teens I'd be inclined to just wait until they were off at uni/left home to move in with anyone else, but mine I still 7 and 4. But I don't feel like you should have to keep your lives totally separate but I also feel you don't need to throw your lives together.