You have children. I don't know if you have boys, girls or a mix - but do you want your son growing up thinking it is OK to have sex with a woman who has said no, and call it a misunderstanding? Do you want your daughter to grow up thinking a man can do this to her, call it a misunderstanding and expect forgiveness?
And how hard is it to understand the word 'No'? When you tell a toddler 'No' (as they reach for something they shouldn't touch, or do something they shouldn't), do you expect them to stop? Yes, you do - because even a toddler knows what 'no' means.
If you stand by your decision to throw him out, it will be painful for a while, but it will be the right thing to do, I am sure.
And if you need more to convince you, imagine it is your dd in your position (or your sister/best friend, if you don't have a dd) - would you tell her to go back to her rapist/let him back into her life?
You are worth so much more than this. You deserve to be loved by a man who knows what that word means - and who knows what 'no' means too.
I am raising my sons to respect women - they all know no means no.