First of all if you have male "mate" friends, can you not get a couple of them to go knock on this neighbour's door?
To be friendly. To be polite. To ask about this person your neighbour saw. But just to let your neighbour know you have male friends who are concerned about your safety.
Obviously the advantage of have male "mates" is that they should be there to help and support you.
Brilliant when you want to move house, or if you want to keep pervert neighbours at a distance.
If these mates aren't up for protecting you, maybe just go check with police.
Say this neighbour makes you feel uncomfortable. You're worried he's targetting and ask for advice.
Police are taking threats against women very seriously at the moment. Every woman.
My friend's 16 year old daughter got harrassed by a man walking home from school. Just asking her to come talk to him and stop talking on her phone. A police report was made online. The police visited their home. And both parents and 16 year old were told to dial 999, if any situation like it happened again.
But you need to widen this out, so interactions and talking about it, so it isn't just you and "pervert" neighbour. You're safer if your friends and/or the police also talk to him. Especially if the police have a word with him. Because he knows he'll be the no.1 suspect if anything happens to you.