This happened to my vulnerable neighbour. He isolated her from her family, then the people in our street who had daily contact with her before he arrived. He then started doing things just like your partner is doing.
She had an accident at home (fell down the stairs) and died a few days later in hospital.
Her partner tried ignoring her family about gaining access for her belongings and, quite frankly, to see what conditions he had her living in. They got the police involved and demanded an investigation.
He tried to do a runner, got caught but there was no evidence of wrongdoing so no charges brought.
The partner decided it would be best to leave this go and, we assume, find another woman to sponge off.
The family had been so close before this man arrived. They were gutted they lost their daughter/sister like this.
I'm not suggesting for a moment that her partner killed her nor that your partner would physically harm you. I don't have to. Mental cruelty is as bad as physical. Get him out.
Call your old friends and tell them what's been going on. Ask them to attend while you ask him to leave. Arrange to have a locksmith there at the same time.
Make sure you have this backup and witnesses.
I think that if you've been ill you should visit the GP or nurse and tell them this man is causing your mental health to deteriorate.
I know you don't feel up to it but you'll feel so much better when he's gone.
However, I would go with the above poster's suggestion to call the police and tell them he's destroying your property and won't leave. If nothing else you have a record of your complaint and can go from there.
Good luck.