A really valid question I think Picadilly. And I totally get what you're saying.
The issue is what we mean by "sex." IMO a massive problem we have, which is a product of our patriarchal society, is that "sex" is seen as "man puts penis in woman and ejaculates." It's seen as something men aim to do, with women acting as the permission-givers and consent-bestowers - they "let" the man do it to them. The subtle but pervasive view is that women are passive in the act of sex and that it's acceptable for them to expect to submit to it, to allow a man to do it to them, in certain circumstances because it's not really about their own wants or needs, or their own pleasure, it's about whether or not they "allow it to happen." This is where many of the rape myths come from IMO - they come from the idea that simply allowing sex to happen by being in a certain place, or dressing a certain way, is enough to mitigate rape. The fact that the woman doesn't want that sex is neither here nor there - her role is not to want or not want, it's to allow or not allow, and in the minds of many people doing certain things allows sex and therefore puts some of the blame on the victim.
Going on that framework, you could argue that yes, a man who doesn't get consent might be just getting sex where he can rather than being out for control or power.
But "sex" in a loving couple and "sex" in rape couldn't be more different. The man who rapes isn't looking for the sex that man in the loving couple is looking for - he is looking for something entirely different. He doesn't get a thrill out of the woman looking in his eyes, showing pleasure, he gets thrill out of a woman who is either passed out or terrified. So there is a difference, a very clear one.
So perhaps it makes more sense to say that sex is something both people enthusiastically participate in and enjoy (the underlying idea of the "only yes is yes" idea KRITIQ mentioned), a shared experience that requires both parties to be entirely on board. Whereas what happens in rape is not sex, it's the man hurting the woman in a very specific way. It's assault, not sex, and behind assault is the desire to control and overpower.
The idea that rape is just sex a man takes without consent shows how fucked up our ideas of sex actually are.
The guy that raped me didn't want sex. He was angry that I wouldn't have sex without a condom and saw me being asleep as a way of getting around my pesky wishes to assert his right to use my body in exactly the way he wanted. We did not have sex. He used his penis to punish me.