Agree with all the people who are telling you to walk away.
We had a boundary dispute with a neighbour who not only wanted our property because of money, but believed she also had a moral right to it because it wasn't fair that we had more room than she did and she'd have done so much more with it than we were doing. And she was nicer than us. Yes, there was a strong element of childishness to go with the obvious greed.
These people sound the same. They've been tending the garden, they were there before you, so in their heads, that makes it theirs.
They probably don't have squatters' rights. Briefly, you have to be using a piece of land, knowing it's not yours, for about 10 years, excluding everyone else, including the owner.
It doesn't count if you enter only at the invitation of the owner. Or, as my nutty neighbour believed, you manage to trick your way on like a vampire whose been invited over the threshold by the dupes next door on the pretext of wanting to look at her guttering, and has taken pictures of herself by their water feature, proving it m'lud 
.
I know that because it formed part, but not the only part, of the dispute. Because of a solicitor's balls-up when we bought, our garden didn't belong to us - or anybody. We found it out but because the garden was entirely enclosed by high walls and the only access was through our house, stupidly didn't get around to doing anything about it.
She discovered that through poring over the Land Registry and thought it was a legal magic bullet which would result in us handing over the keys not only to the garden but to the entire fucking house. Yes, she's that mad.
She sold up after it was settled in our favour at a property tribunal. If I'd have been a stranger her behaviour at the tribunal would have been funny. But it was me and it was four days of worry plus two week's nervous wait for the decision. That was on top of six years of expense and stress. Oh and trespass, malicious calls to the police, damage and petty theft.
There was a period of about two years where DH and I talked about that mad fucking bitch every day. I didn't want to meet her in the street and I'm not a wimp. It's four years on and if I see someone who looks like her it makes me look twice.
Of course, your neighbours might be entirely different to mine. But the signs are there 
You can see by the length of my post that I'm still working through my issues 