He might not mean any harm, but he's making you feel uncomfortable.
It's possible that this is all a coincidence, he means no harm, and you've been rude.
But you don't owe him politeness even if he is harmless. You don't want to speak to him outside of your professional role and he needs to respect that.
It's also possible he's following you on purpose for whatever reason, and your instincts seem to be telling you that this is the case. We're always telling women not to ignore their instincts, so please listen to yours.
If you've seen him in a professional role, speak to your manager or your professional body for advice. You can almost certainly have ask for him to be seen or treated by someone else.
And you will have made someone else aware of his behaviour and the effect it's having on you. It's important that they know.
And like Pomp I am sick of women being told to change their behaviour or be polite to someone who is bothering them. It doesn't matter if he only wants to say hello or have a harmless chat. She doesn't, and he's made her feel uneasy and uncomfortable in her daily routine.
Me Too wasn't only about rape and sexual assault, it was about all degrees of harassment of women by men, including verbal harassment.
I described what happened to me as 'nothing out of the ordinary', which was a depressing enough thought as it was.
But Me Too made me realise that the earliest time I was verbally harassed and expected to be polite about it was by an older boy when I was seven years old and after verbally harassing me he complained to a grown up that I was being mean to him because I was trying to avoid him.
Of course, he didn't tell her what he'd said to me, and I didn't have the words to explain myself. And that grown up made me go and be polite to him.
It shouldn't have happened then and it shouldn't be happening now. The OP is being made to feel uncomfortable and she doesn't owe politeness in return.