There is something about his response and what you've written that leads me to think that this was a conscious transgression of your boundaries, and I have a suspicion that he's the type of person that doesn't think that removing a condom deliberately during sex isn't rape. Or, if he does think it's something that men shouldn't do, then he wouldn't regard it as seriously as he should.
Even if you don't want to, or don't class this as rape, or are confused about it, I say listen to your instinct.
Not all forms of abuse are to do with anger and deliberate cruelty or torture. It doesn't mean they spend their time angrily scheming about what to do to you.
Some abuse is just a form of extreme selfishness, lack of caring, ruthlessness and coldness. It can come from an unhealthy power dynamic, where a man has a casual attitude towards a woman's consent with an entitled attitude to her body. It's when men don't listen to women, don't take them seriously and things like this as 'just what men do'. Your sexual boundaries and minor inconveniences and something for them to decide on your behalf.
These are the type of horrible men who have unprotected sex with women on the assumption she'll be fine with having an abortion. They don't really care about her or her body or wellbeing. It's all fine, shit happens, no need to take it so seriously, and something she should be sorting out as a good little women for him.
It's important than women trust their partner 100% and that she trusts them to look after her when she's vulnerable or old. If the answer is 'no', 'sort of' or 'mostly', then reconsider your future with that man.