I, too, noticed the fobbing-off with "you're obsessed..." and that disgusts me almost more than his career woman comments. The "non-tactile" thing is a red flag as well, but see below.
Sorry, OP, I'm with the ranks of posters who think he's got somebody else (or planning to) or, alternatively, a porn addiction. Either can result in a decreased sex drive with the real, breathing woman in one's life.
But I also agree with cleammyhouse, in that it almost doesn't matter.
Bottom line, and I'd say this to anybody: why does he get to decide? Answer: he doesn't, any more than you do. Vote with your feet.
Finally, you're wondering why he doesn't want you to leave? Because he wants you to obey, that's why.
Between this and some of the other things you mentioned, I'm sensing there's a lot more red flags in your "rollercoaster" relationship you haven't mentioned. Read up about abusive men; I'm sorry to say you'll probably have your eyes opened.