I expect you've found these comments quite surprising OP. You've only just started feeling annoyed and here are lots of people saying he's plainly selfish, thoughtless and not a good partner. It must seem quite a leap from 'a bit annoyed' to 'take his key and lay down the law' and even 'get rid'.
To get from one to the other, just think through how he's going to react when you say 'I need some time to myself, so I need you to ask before you come over, respect my preference that we see each other 3 or 4 times a week, not every night and respect my wishes when I say I want an evening alone'.
I am willing to bet that he will in effect turn on the tears; go all self-pitying and manipulative and say that he thought you loved him and would always want to spend time with him, he would always prefer to be with you and he's sad that you don't feel the same way. And what's the point of the relationship if you don't really want to be with him, he thought you loved him, he was hoping you'd move in together but what's the point if you just see him as a casual boyfriend.
Then you'll get upset, say you're sorry and cave.
The important thing to remember, is that just because you can't think of the words to express your feelings convincingly during an argument, does not mean your feelings are not real, valid and very, very relevant.
So, prepare your thoughts in advance. Be proactive. What do you want and need? It sounds as though you want the relationship with him to stand the best possible chance of success. That means taking it at the right pace, finding out that each stage works for both of you before progressing to the next, not acting in haste to repent at leisure. You need some time to yourself, because you do and so that the time spent with him feels special. By imposing himself when unwanted, preventing you from getting on with other elements of your life, he risks making himself a burden, not a pleasure.
I suspect he hasn't even considered that you have aspects of your life and needs that exist independently of him. He feels comfy at your house, therefore you should feel happy to have him there. He may just lack empathy and insight generally, or particularly with regard to women, who he probably, like many men, sees as being there to provide things that men want, rather than actual autonomous people.