You don't need to do silly things like swap the key he has. Everything you write tells us you know this is a dead-end relationship that is not satisfactory for you on any level. You know the answer, tell him it's over. Do him a favour and tell him why. If necessary if you think he might turn nasty, have someone over, even if in another room. And if he simply cries, well when he brings his washing back to his mother, maybe she can find him a hanky. You can't justify sticking with something you longer want because someone might be, or purport to be, upset. That's no way to live.