He arranges his days in to be around you
He waits around to be at lunch with you
He assumes lifts will be available with you.
He is monopolising your time and attention, and seems to think this is ok, and to expect it from you.
All this together suggests he is making a great effort to be with you because he has formed a fixation with you - and these can turn bad.
It's been going on for a while and has become a habit, the norm, for him.
Women are socialised to be kind, be nice, be quiet, be accommodating.
You need to speak to him clearly about this.
Do it by email if you need to.
Explain you don't feel comfortable spending so much time with him, that you require him to give you space.
Arrange things that show you mean what you say - lunch arrangements with other colleagues, work days that suit you, WFH if you want to. Concrete control over what you do with your time rather than stories.
Then if it persists, you can go to HR with the evidence you had told him no.
It's a shame if HR would require you to have had the courage to deal with the difficult situation first, because so many don't have that, and end up in unpleasant circumstances.