Okay I get you.
It's not going to be easy. But the police will come to stop a breach of the peace but no, I don't think they will force him to give the key back and even if they did who's to say he hasn't already cut another key?
also the police are not relationship helpers, and while they will do things like this you seem to expect more of them than they are likely to offer.
So yes they will 100% turn up quickly mob handed and eject a rowdy bloke from your house, like I say, I've had this a few times (man I was a mess) and they will even come with you to collect your belongings but when they did they abandoned part way through after being very rude and rushing me. I was not able to get everything and they were going to leave me with my ex's friend alone in the house.
So while 'can you come and prevent a breach of the peace?' is likely to be a yes, 'can you come and force my ex to give me my key' that's likely to be a no.
This is why I said just change the locks, then if he makes a fuss outside you're hitting off to a great start because the police will come and will arrest him if he won't leave, and also you've then got clear grounds for a non-molestation order which then gives you clear grounds to call the police every single time he tries to come near you, and this increases the likelihood of this all being over.
Whereas if you ask the police to come and try to break up with him, he gives you a sob story and you end up saying okay then stay, the police won't help you stick to your guns, they'll just leave and will not take you seriously in future.
So I really urge you to take action and change the locks instead of breaking up with him in person and risking him talking you round and risking losing help from police part way through.
I'll give you an example. I was in a station with my ex, he was pushing me and calling me names and a lady who was a police officer ended up coming over and making it clear to him he better leave me alone or she would get very angry.
I said one word, I said 'wait' and she looked at me and went 'what? you want to be with him do you?' I said 'yes' and at that moment she washed her hands of me, walked away, and gave no more fucks about my situation that I had clearly got myself into.
They like arresting people but they aren't relationship counsellors or bouncers.