Eye contact: (info from a brain scientist, so don't blame me if it's dodgy. And I will be generalising, as we're all different (as we can see from some responses above)).
In most people, eyes make contact with other eyes....and it might be a predator about to eat us...so their brains send the signal to the most ancient bit of the brain, the amygdala (wiggly bit in the middle of the brain). It identifies whether it's a predator or not by asking the other bits...and if it is, it presses the "Hell, Panic! Run!" button and the brain prepares for extreme frightening attack. If the other bits of the brain say those are the eyes and face of a friend, then it relaxes, and lets the people-bit of the brain take over with its whole "Hi, welcome, friend!" sequence, including the eyebrow raise greeting (people always do this when they see a friend - watch it!), the smile, instant recall of info about them etc.
In our brains, it sends the eye-contact signal to the amygdala, and there it stays, pressing the "Hell, Panic! Run!" button the entire time we're looking into people's eyes. So, even if we know it's a friend, our brains are stuck with a bit that is pressing the "Run!! For goodness sakes run!!" button continually.
To talk to you, we have to look away. It's not rudeness. Yes, we can be trained to look into eyes and just be scared witless doing so and learn to carry out a conversation whilst scared witless, but for the life of me I can't think what we'd gain from that. People don't complain that Blind people can't make eye contact, for example.
So, that's the 'why'. Exceptions apply.