When I try to, in a jokey swipe his hand away from my intimate areas, he brings it straight back. I'm not sure if I'm imagining it, but he seems to be more forceful with his hand movements, like he is mad at me.
You're not imagining it. You are clearly expressing 'no' and that's going to make him angry, because he doesn't think you have the right to say no to him. Because he is abusive.
When I've tried to put him off popping over, he gets mad at me, again, he seems to be a bit "rough" with me.
He is trying to frighten you so he can maintain control over you. He sounds scary.
I don't want to risk it
Completely understandable. You need some back up, some support, a plan. Again, get in touch with Women's Aid. But also think about what you could do to keep him away. Could you make sure there is always someone else in the room with you, so he can never see you alone? If that's not possible, is there a way to not allow him in your home at all? Do you think he's likely to give up if he can't get to you?
I know you might find it a bit of an intense thought, but could you consider reporting him to the police? What he's doing to you is against the law.