I think this is all very one sided from what you have said. You clearly had big feelings for him at the time and still do when your paths cross, but other than a few drunken fumbles (as a pp said - standard young person stuff), it doesn't sound like he has ever really given you any indication that he has actual 'feelings' for you, or that he would want to pursue any kind of relationship.
In the work situation, if you had never met him before, would you think he had an attraction to you? It doesn't really sound like it? Has he spoken to you personally at all? Sort of 'really nice to see you again', type of thing?
This kind of happened to me in reverse. I had a fling with a guy I worked with in my early 20s, similar stuff - we would end up together at the end of nights out but it never really went anywhere and it messed with my head so I had to stop it. I was infatuated with him, obsessed really. It wasn't good and thankfully he left the work shortly after. It was all forgotten and I moved on.
Anyway we bumped into each other at a couple of work things years later, all friendly etc. Then at the end of that project, he took me aside and confessed all these feelings he still had for me which had been reignited again. It was really awkward, I was so taken aback because it just wasn't where I was at anymore and although I let him down gently, it was quite embarrassing.
My point is, he may have moved on while you haven't. So any declarations of your feelings might take him very much by surprise and not necessarily in a good way.