I’m guessing that this older male colleague doesn’t seek out his younger male colleagues, compliment them, ask them out for a drink, and make comments on their ID pass photo.
You are in a difficult position because this is the workplace and you can’t easily tell a bothersome older male colleague to leave you alone.
This colleague shows signs of being more than just slightly bothersome though. The latest incident - in which he clearly made a thinly-veiled accusation of lying - shows that he is not a decent man. The comment he made about you in front of another colleague shows that he will seek to undermine you and trash your reputation given a chance.
I think you need to be on your guard. Women who live alone often find themselves targeted for unwanted attention by a male neighbour who feels entitled to their time and attention. There’s plenty of threads on here with examples. Those kinds of men are likely to do the same in the workplace.
Don’t spend any time alone with him. I’d be surprised if him suddenly appearing when you were alone and on your way to your office was actually a coincidence.
If your employer has HR policies on workplace behaviour, familiarise yourself with them. Make a written note of every interaction he has with you, so that you’ve got a record in case he decides to punish your lack of interest in him by making an unfounded complaint against you. If you’ve got a confidential employee support service in your workplace, get in touch with them. Hopefully he’ll leave you alone, but at some point you may need to have a confidential word with your line manager or trade union rep if you have one. If you’re not a member of a union, I’d suggest joining one now in case this escalates in the future.
There’s been quite a few “you’re making a fuss about nothing” comments on this thread. There have also been several posters pointing out the red flags. Your sense of confusion and discomfort with his behaviour, which is why you posted, is because of those red flags. He is pursuing you, and using “plausible deniability” tactics in the way he’s approaching his pursuit of you, so that people will side with him rather than you if this escalates / goes ‘public’ in the future.
And if anyone says “aww, he’s just being friendly, you’re being mean/ precious / whatever OP”, I refer back to my first para. Is he behaving in this way to younger male colleagues? I very much doubt it.