It's more that all manner of non related things could be used against you.
Anything not related to the rape cannot be used against you in court about the rape. It would be inadmissible. They can only use relevant evidence. What they are looking for is evidence of consent to sex. They don’t care about anything else for the rape case.
They’d have to start a separate case and charges against you, if you’d committed some other crime and was stupid enough to have evidence of it on your phone. Say, hate speech on 4chan or using BBC iplayer without a TV license.