When ex and I split, we owned two houses between us, next door to each other. We each had both sets of keys.
a couple of months after we split, I went to watch DS in a concert, locking the door to the house I lived in. Shortly after I had left, he went into 'my' house, using his key, with the DD's to 'walk the dog' (coming from the man who would get in the van to do 200yrds up the road)- who I had just walked. I found this out a week later, from DD's. He didn't ask, he didn't let me know afterwards, he had no belongings left in the house, DD's had taken all necessary clothes etc. for an overnight stay, so he had no reason to be in there, other than because he could.
I changed the locks. A couple of days later it appeared as the someone had tried to get in through the front door when. Had been out for a morning (it is a really tight fit, so if you turn the handle it clicks, and you have to pull it hard to close it properly). And he had changed the padlock on the garage attached to the house I lived in. He refused to give me a copy of the garage key until he had taken all of his things out of it, something that took him a further 7 months and a couple of sessions of having him shout at me when asked when it would be done.
We have been separated now for 19 months, we have signed the transfer of deeds, but it isn't completed yet and last week I actually got the key to one of the sheds that will be mine after he finally shifted yet more of his stuff out of it.
Legally, if he has his name on the deeds he should be allowed entrance. You need to buy him out ASAP. In the meantime, as my WA support worker suggested... Loose our keys, ask him to knock on the door, not just let himself in, and he should be none the wiser.