That's why I'd leave it, it takes time to get your head around the massive life change of breaking up, its not impossible. Its a challenge that you can overcome with support and understanding from others.
My ex, brilliant with money, worked hard, wouldn't cheat, one night out a fortnight (but would override my nights out on the inbetween weeks), he admitted he had a problem with being controlling. He used to tell me how intelligent he was (frequently) but since splitting I have found that I'm the sharpest knife in the drawer after all.
But all that, did not make up for the fact we were not a match. And even though he was 'good' with our eldest child (calm, full of praise etc) he didn't actually involve himself in family life. He didn't want both children together (too stressful) didn't want to go on family days out (stressful), holidays (stressful), wanted them in bed by 7pm exactly even though eldest was easily able to stay up later, he felt his evening break was more important than her sitting awake unable to sleep for an hour or more. He just was not a part of the family unit, it was like he circled like a satellite outside of it. He also got threatening at the end and I having mentally prepared myself for the break up, got finances sorted and in place we (eventually) separated. He has been a dreadful ex, I didn't think he would be as bad as he has been. But never for one moment do I doubt I have done the right thing for myself and the children. His behaviour since has spelled out just what a person he is and has done nothing but put himself and his needs over and above the children's. We are better off with a big space between us and him.
But its a big step to take and you need to be sure in your own mind before you make a decision. Try to make it work for your own peace of mind. Set down boundaries. Say what you want or need. In my case my ex compromised (actually I think he was bloody furious inside, but as he had become aware that he could lose the house if he split with me he 'did as he was told'...I just asked nicely for him to wash the dishes...) and he tried for a few months (flowers from the garage slammed down on table, on schedule, once a week, his idea...) then he caved and started becoming his usual moody self...like a miserable grey cloud wandering around the house...it became obvious things could not continue as they were. Turns out he didn't like me being assertive and expecting him to treat me as an equal. He preferred a doormat who did as they were told. He still blames me for the break up...and tells the children he 'tried everything'