Moo, just leaving aside the issue of the definition of sexual assault for just one minute.....I'm not sure if you mean that he is constantly attempting to penetrate you with his finger and you are finding it bloody irritating to constantly tell him to get his hand off there, or whether he tries to do it and you let him do it but you are not happy with him doing it and feel turned off and annoyed. But just think about both of these situations and how it makes you feel during sex:
In the first scenario you might well be tensing up as his hand heads down that way. It starts to play on your mind: oh god, he is going to try that one again." You get prepared to push him away or tell him verbally not to do it and then the time comes where you have to stop him and he might well tut or huff and then carries on to PIV sex. Yes, he might not technically be penetrating you in this scenario but my god, how disrespectful and annoying ad distasteful is it that you have to go through the possibility of this scenario every single bloody time you start to have sex? It must surely at the very least put you off the rest of your sex session and then are you just going along with the rest of it to satisfy him? If this is what is happening in your case, OP, then you have to realise that it is not normal to feel tense and apprehensive every time you start to have sex with your partner. It is NOT normal to have to repeat each time that he must stop what he is doing. Just how do you see your sex life continuing in this vein for much longer before you are either a) put off having sex with him in any way, shape or form or b) you decide to give up protesting at it and just "lie back and think of England" which is basically what scenario 2 is.
In scenario 2, you might be having sex, his had wanders down there, you tell him you don't want his finger up there (having told him many times before) and yet - he does it anyway. You can't possibly be enjoying it at this point, OP, so what exactly are you getting out of it? Are you just wanting to get it over with so that he will leave you alone for the rest of the night? Is that how you believe happy sex lives between couples should go? Really?
So just completely leaving out the whole "definition of sexual assault", OP, as it might be too distressing a concept to deal with right now, if I were you I would have a bloody good think about what is actually happening here and whether you think it is an acceptable way for a loving man to behave towards his partner. If what is happening is similar to either of the 2 scenarios described above then I would have a SERIOUS, long hard think about what you are going to do about it. Because neither of them are normal. You might kid yourself that Scenario 1 is somehow acceptable to you for the moment. But can you see that it couldn't stay that way? Your feelings, or his actions are only going to get stronger. A scary thought. And then scenario 1 becomes scenario 2 before you have even realised it is happening.
If you're already at scenario 2 then I really think you need to get yourself some professional counselling to help you deal with it.
Best of luck.