I am not asking all adult men to carry the lable rapist. I haven't said this, implied this or thought this. The very closest I have come to this is saying that a lot of women have experienced assault, abuse, harrassment or intrusion in public, from adult men, and therefore that adult men modify their behaviour very slightly to prevent people from feeling uncomfortable.
I will give you another analogy to see if this makes it any clearer.
I love packets of assorted sweets, but there are only two flavours in that packet I actually like. When I work from home, I often go out for a walk between three and four o'clock and may buy a packet of sweets on my way.
Therefore, on a fairly regular basis, I find myself with a big packet of sweets I don't want as children are walking back from the local primary and secondary schools.
Do I think "offering these sweets is harmless. I know I'm harmless, so I'm going to go up to people and offer my sweets?"
No. Because although I know I am harmless, I also don't want the children in question to feel threatened or uncomfortable. I am not forced by society to think of myself as a potential child abductor, and go around wearing it as a label. Instead, I just think about how my behaviour may appear to others and, y'know, modify it.
I don't disagree that women and children face a greater threat inside the home than outside it. But I don't see its relevance - they are very different threats, with different causes and responses.
And outside thinking of this in terms of rape and murder, women and girls experience encroachment on their space in public every day. From my examples to wolf wistling, "smile love", men trying to talk to you at a bus stop after dark, men spreading their legs on the tube so far that you barely have any seat.* What's so damn wrong with asking men not to perpetuate that encroachment?
*Maybe we wouldn't perceive all of these examples as encroachment if we lived in a touchy feely, strangers start deep and meaningful conversations with you on the street sort of societ, but we don't. We form a society which values space in public (and by "we" I mean humans in general, not just the English). This is blindingly obvious to more or less everyone who doesn't have an impairment such as autism. Including - gasp - adult men. So what is so damn wrong with asking them to think about how encroachment is perceived in a society where they still enjoy a very high degree of privilege and asking them to, y'know, show a little consideration?