OP, I have been in your situation, a few other people have said the same and I bet there's more reading this that have too.
It is rape, just because you didn't say no or fight back, you were asleep. By legal definition, he raped you.
When I was younger and 13 weeks pregnant with my son I was staying with a friend and woke up to find her partner doing something to me. All I could think of was, what if he hurts me and I lose the baby? Also, I was sleeping in the bottom bunk and his own daughter was in the top. I don't want to go into detail, too upsetting for me, for others reading and possibly outing.
The next morning I got up and got ready to leave for work, he acted like normal, he thought I was asleep and never knew. It was awful.
8 years later I meet someone else and confide in him what happened. I woke up one night to find him raping me, and this time I did say 'what the fuck are you doing?' A different response, but it was still rape.
I never reported the first one. The second one I did. I now work for an organisation that helps rape victims get justice, if that's what they want.
For you now, the most important thing is to remember that no matter what others may have written on here, it did happen, he's in the wrong and you froze - not your fault. Whether you go to the police or not is completely your decision. For today, you just need to find a way to get him to go - only you know your circumstances and what will work. Some have already made good suggestions.
I know you're shocked that someone you thought was so wonderful has done the unthinkable. That's how they trap us - by being so great at the beginning. None of this is your fault, now you just have to find a way to deal with it.