Tbh it doesn't surprise me in the least that he's a psychiatrist, they are as a general rule the most unpleasant, emotionally unintelligent, power-crazed, narcissistic and misogynistic bunch I've ever had the misfortune to work alongside! (But then I'm a lovely, warm, fuzzy counsellor
)
He's right about one thing though, you do have crap boundaries! You're allowing him to treat you however he likes. So what if he threatens to 'leave'?! Leave what exactly, the room? The job? The uk? So let him!
You need to tell your line mgr or someone senior what is going on. He is harassing you and it's completely unacceptable. Be honest that you slept with him (unless your workplace has strict policies that will mean you get in trouble) and that now the relationship has ended that he won't leave you alone, that he comes in early/stays late to bother you, distracts you from your work, makes comments about your weight/sexual comments which are unacceptable.
At the very least someone should have a word with him, and this may be enough to get him to stop and back off. You need to stop feeding him, just ignore him totally. If he turns up at your desk say 'I've got work to do, please go away' or some such. You need to be firm and consistent. Use the broken record technique.
And if that all sounds completely impossible because you feel you can't or you don't want to upset him then think about getting yourself some therapy as you may need some professional help with moving on from your past unhealthy relationships and learning to set better boundaries/developing more self-respect.
Good luck! X