I agree with the others that leaving would be the best option - you don't even have to really say why if you're not comfortable doing so, you could just say "I explained to you that I was not prepared to take on a large workload and long hours and yet that is what you've given me."
However, if you are truly set on not leaving (and part of me understands your reasons and I'd feel the same I think) you do have options.
I have funny rules about being touched too based on previous experiences with some not-very-nice men. It's a bit odd in that there's some people who can hug me or whatever and I'm not bothered in the slightest but there's others where I tense up if they even innocently brush past me. I've never worked out what is in my mind that differentiates between them, so I've just learnt to accept that that's the way I am now.
I work somewhere which most people would struggle with, it's very un-PC and not exactly professional which I don't actually mind for the most part. But there's a lot of hugging or just bumping into each other or whatever while people are having a laugh while working and one of my colleague got very upset that I would hug and join in with certain people but picked up on the fact I couldn't do it with them and had just said to them "it's not personal I just don't like being touched" and they were understandably offended and had a word with our manager about it.
I don't have a particularly good relationship with the manager in question and it was excruciating to do it but I sat and explained to him briefly about my past and why I was the way he was and, in fairness to him, he's been great about it ever since. He handled the situation with the other colleague and he's always very aware if I am in a situation where someone hugs me/brushes past me/touches me to move me out the way and checks I'm ok.
Sometimes I think people (and IME) men especially don't think if you say "I just don't like being touched." So while it should be enough to say that, I think a lot of people will just view you as standoffish which is why I chose to be very upfront about it. I didn't go into detail I just said "there's been events in my life that have meant I now only trust certain people to touch me in any way, I know it's mostly innocent to them (though in your boss' instance, not so much, it's definitely inappropriate) but I can't help my reaction to it" and then they get it.
You shouldn't have to do it, and as I said first off, I do think it would be best for you to leave. But if you are dead set on staying I'd consider saying something like that just because unless he's a complete and utter arsehole he'd probably listen to that and actually hear what you're saying rather than just thinking you were being a bit funny about it.