Even if he has paid rent and a solicitor deems that he has an "interest in the property" that doesn't give him the right to a property he legally has no current share in. The law is quite clear on this.
He can go to a solicitor and pay for them to bring a case against you stating that he has an active interest in the property for x amount, but that's about it. You have every legal right to change the locks on YOUR house and also tell him that if he continues to hover around the property you will get a non-molestation order in place, and if he breaches it he will be prosecuted.
I would do the following:
Get a solicitor to write to him and state that the locks are going to be changed, that he has no legal right to enter the property and that his things have been removed. That if there is anything missing from his items he is to contact them (the solicitor) and you will ensure the items are provided for him. He is not to contact you under any circumstances.
That if he has any issues with the situation he should instruct legal counsel, and that if he attempts to break into the property or refuses to leave, a non-molestation order will be sought and the police will be involved.
Hopefully that will work and he will understand you are serious.
Before the letter is sent, change the locks while he is not there. If he becomes violent, call the police immediately and inform them of the situation, have people there with you to ensure there is backup and install a camera doorbell as well.
I might seem ott but having been in a situation with someone where i had to threaten a non-molestation order, i understand how scary it is. I co-owned with my ex and he refused to leave and i couldn't get him out due to him being a joint owner, but I did refuse to pay the mortgage and threatened him with occupational rent and a non-mol if he continued to contact me and threaten me for refusing to pay. You can win this.