I hear you @meggy96 As many posters have said, trust your instincts. Women are never wrong. Ignore the 'poor menz' naysayers. They are obviously men.
I am experiencing something similar right now. I go for walks every day - 2-3 miles, and most of the time it's around my village. (It's around 1.5 miles square.) I walk through the village, around the village, in the woodlands, near the river, near the canal etc...
For this past 6 months, I keep seeing this man, I'd say 3 in 5 times I go out. I am middle aged and he is about 10 years older. It's becoming really fucking weird now, and too often to be a coincidence. I don't see ANYONE else this often on my walks. Some people I see once or twice a month walking around the village, if that.
It's mostly when I walk down the high street (through the village,) to the horsefields, rather than the opposite way into a little woodland and across a field to another little woodland. OR when I walk out of the village and go towards the river. So it's like he lives in the high street somewhere (or off one of the little 4 or 5 cul de sacs coming off the high street,) and he is spotting me, and grabbing his coat, putting his shoes on, and coming out.
Some days, after 5-10 minutes, I see him a about 300 yards behind me, and sometimes, I have been walking 10 minutes or so, and he appears at the bottom of the road that I am on (but he has gone round the other way in the hope of bumping into me.) He has spoken a number of times and kept me chatting in the lanes, and asked intrusive questions, and now I actively avoid speaking to him because he gets on my nerves.
But yeah, he gives me the collywobbles to be honest. Some may call me paranoid, but as I say (and loads others have) it's an instinct. I see other men on my walks too, some quite often (not as often as him!) and I feel nothing. They just smile, and say 'hello, nice weather for the time of year lalalalala...' I just say hello back and chat for a minute sometimes, and don't feel threatened or bothered. But THIS man creeps me out. I can't articulate it very well, and I don't know why, but he does.
Yesterday I went for a walk around the village, and had been out around 15-18 minutes. I was coming back home up the high street, and he just popped out of a cul de sac road opposite to me. I saw him out of the corner of my eye, knew it was him, and didn't make eye contact. I just got my phone out of my pocket, and pretended to phone someone.
Still on the opposite side of the road, he started walking ahead of me. I went into into the bus stop, and kind of hid away, and came back out 2-3 minutes later, and he had stopped walking and was looking back! So I kept on 'chatting' on the phone, and carried on walking. I stopped again after 100 yards or so, and looked back towards where I had come from and pretended to take a photo. I looked back a minute later - and he had stopped again, and was looking down the road at me! 
No-one is going to tell me this is a coincidence. It's weird. HE is weird. He is freaking me out. And I don't like it. As I said, I don't feel like this about any other man, just him. Like a few people have said @meggy96 trust your instincts. If someone gives you the heebee jeebies - there is probably a good reason for it.
I have no idea why this man keeps appearing some 3 in 5 times I go for a walk around my village, why he walks the same way as me, (and trying to accidentally bump into me,) and why he keeps stopping when I stop. But it is grating on me now. Maybe he is just lonely and wants to chat to me. (I have never seen him with anyone else, ever.) Well I don't want to chat to him, so he can fuck off.