Yes, sounds like a weirdo and you did exactly the right thing. I think some men follow women on purpose to frighten them. They don't intend to actually do anything but enjoy the powerful feeling of knowing that they could if they wanted to and the woman doesn't know that. Or he was actually intending to harm you. Or he was just very strange and confused. No way to know, but always listen to your instincts.
I was followed in a city centre a few years ago when I was with another mum and our young children. No idea how long he was doing it before I noticed, but he looked like a completely normal, handsome, smartly dressed young man. Blended in entirely with all the other suited people around us and absolutely gave me the creeps.
My child's shoelace had come undone so I had to suddenly stop in a fairly busy area to tie it up. I noticed that he stopped as soon as I did, but he wasn't close enough to us to have needed to stop. He could easily have carried on and gone around us. I thought that was odd, but I smiled at him in a 'oops, sorry for inconveniencing you' way, and he returned the smile but just stood there, still smiling and watching us. Him standing there watching me crouched down tying was really unnerving so I kept looking at him to see whether he was leaving. He realised that he couldn't stand there forever, so he pretended to carry on walking, but was watching us in window reflections the whole time. When he thought I wasn't looking, he doubled back and waited a little bit behind us again. I looked right at him and he crossed over the road, walked over there for a bit, then crossed back again when he thought I wasn't looking. I don't know what the hell was going on with him but I didn't want to find out. I 'accidentally' walked right into the middle of a group of people and he ended up walking away in the end. The other mum hadn't even noticed! That blew my mind.
Ignore others saying 'ooh men can't even look at women anymore, ooh men can't even be lost jogging in the woods anymore'. You didn't offend or insult anybody, you just listened to your instincts and kept yourself safe. We all know exactly the looks and behaviours that creepy men get away with in public because they can be innocently explained away by people who weren't there. I believe staring has been added to the list which makes up sexual harassment on public transport. Because we know the difference between making normal eye contact with a man and a man behaving in an intimidating and threatening manner. I believe you and I would have felt exactly the same.