YES! You can and you must leave him. He has raped you on multiple times, that is more than enough justification. Look at it another way: at the moment, those children live with a sex offender who is a serious threat to their mother.
His sexually abusive nature will not exist in a vacuum alongside the rest of his behaviour and personality, he'll be abusive and controlling and unpleasant and selfish and disrespectful of the needs, wants, rights and boundaries of others in other ways, even if those are not sexual in any way. Living with someone who abuses their mother, even if they don't see the worst of it, ie the rapes, is abuse of them, mentally and emotionally. He is not a good father. A good father does not rape a child's mother. He is not fit to live with anyone, never mind kids.
Your right not to be raped trumps anything else. You cannot decide to tolerate sexual assault for anyone else's wellbeing. Divorce is not the worst thing that could happen to those kids - the worst thing that could happen to them is already a reality, because their mother is being raped, abused and gaslighted by their father. It's already the most heartbreaking thing ever.
He is being okay now, because he knows you know it was rape and he knows he's pushed his luck too far, and is trying to reel you back in and get you all confused about the truth. All abusers are nice sometimes. And besides, he is learning that he can rape you and get away with it, so that's fulfilling most of his need to abuse.
A final thought: how would you feel if you found out that your father had raped your mother repeatedly, but she'd stayed for the sake of you, her child (and any siblings you might have)? How horrified, guilty, and appalled would you feel?!