Op - you need to find a way to leave, not keep looking for justifications not to leave.
You don't love him, you don't find him fit enough to overlook not loving him to have sex with him. You are miserable with him.
Too many people feel they need a good enough reason to leave a failed relationship, cheating or abuse, but not loving or liking the other person is enough.
It's ok to start making end of relationship plans. If he won't leave nicely, have you seen a solicitor about your options?
You are a high risk of an affair right now - you aren't happy, don't want to be with him, don't feel you have good enough reason to force a split when he won't give you a nice cosy amcable split, (and are sexually frustrated), you would fall for the first bloke who showed you affection and convince yourself you are a Romeo and Juliet, not because you really love the new man, but because he'd be a clear exit route.
Find an exit route that's less messy. You might have to be the bad guy.