You're the double of his ex? Then be doubly beware. His behaviour as you've described it seems slightly cruel; he's cultivated your interest, then politely dismissed it, yet keeps hanging around you at work and contacting you outside of it. I'd guess he gets off on the emotional turmoil he's stirring in you, perhaps as revenge on his ex-wife.
As a fellow limerent, I feel for you. It's like your brain has been hijacked. (In my case it's lifelong and caused by ADHD.) So here are my management tips:
You must override your lizard brain. If your thoughts stray to him, count backwards: "5-4-3-2-1, nope, not thinking about that, piss off, get lost." The backwards count engages your pre-frontal cortex, your centre of reason.
You might also have a corresponding surge in physical energy. So use it. Go run, cycle, learn ballet, or just walk for miles with an audiobook. You'll feel fitter and more in control.
Tell him (whether it's true or not), "I'd rather you didn't contact me outside of work. My social life has become very busy, so I won't have time to chat." Then smother a smile and make him wonder what you've been up to. And keep your private life private. No more cosy confidences with him.
Normally I wouldn't discourage someone from pursuing a crush, but this guy is giving me the serious ick. I think he's using you to embiggen his limp self-esteem, OP.