He's being sexist. And it's pretty telling about what he actually thinks of you.
"I trust you, I just don't trust him" tends to mean "I don't like other men sniffing around my property".
Because actually, if he literally trusted you but not the other man, then he's calling the other guy a rapist. Because if he trusts you then he knows you would turn down any advances and MOST people would then say "Oh crap, sorry, totally misread the signals there, let's forget this ever happened!" If he doesn't trust the other man then he's saying that he thinks he wouldn't do that, and would continue/insist anyway.
Either that or he's saying "You wouldn't do anything on purpose, but you're pretty weak and stupid, so you might not notice if he was manipulating you into sex".
The first is a red herring - if he genuinely felt worried that this man was predatory and/or potentially a rapist he would be worried FOR you, not worried about any attraction developing.
The second means that a, he doesn't REALLY trust you, and b, he has a pretty low opinion of you/of women in general. And also possibly implies that he believes all men tend to be a bit manipulative and use coercion to get women to sleep with them/go out with them, which would mean that's what he does.
The third scenario of course is that he has a problem with other people fancying you, which makes him out to be pretty controlling, since he can't control everybody else's thoughts in the world, and there's nothing wrong with somebody else fancying you, as long of course as you don't act on it. This is the scenario, IME, which is "My woman = my property, nobody else is allowed her". Controlling.