He should be giving you 24 hours notice and requesting your permission to enter the property.
I would remind him of that.
It is categorically not okay for him to turn up when he likes and enter the property.
Feuds with neighbours etc are nothing to do with you and the property is currently yours and not for him to have access to.
Of course he shouldn’t be taking it upon himself to enter the property or to touch your bleach and your bread or anything else.
It sounds like he has trouble with boundaries and in either grasping or thinking the law applies to him.
I assume you have all the correct contracts, right to rent booklet, gas cert etc. Regardless, I would email or write to him telling him that you are not comfortable with him coming round on a regular basis. In future as the law requires you expect 24 hours notice of intent and reason and that you okay the time. Point out his visits should be infrequent.
If you have allowed or even encouraged this ‘agreement’ whereby he turns up randomly and enters the property, you need to make clear that you’ve rethought it and are not comfortable with it.
Your commentary on the matter is very confusing and vague. I wonder if the arrangement you have with him is very informal and little if anything is written down. This is. It the way it should be and having everything done legally and it all written down protects you both. Notice of visits and the importance of you having the right to ‘quiet enjoyment’ of the property you pay for, should all be written down to avoid any doubt.