As others have said, they need to have taken possession of the land without your permission (or the previous owner's permission if relevant) for either the last 10 or 12 years depending on if you bought your property before or after 1990.
Can I ask, when did you buy your property?
If you bought it at any time from 1990 then it will be registered with the Land Registry and they would notify you if there is any claim for adverse possession. I think that you sort of implied this was the case in your OP:-
"they've never applied for it with Land Registry and we would have rejected if they had applied"
It seems as though you have a "neighbour from hell".
Frankly I would ignore them as much as possible and/or move away.
This is an excellent explanation f adverse possession:-
www.gov.uk/government/publications/adverse-possession-of-registered-land/practice-guide-4-adverse-possession-of-registered-land
The meaning of adverse possession is that:-
"Factual possession signifies an appropriate degree of physical control. It must be a single and [exclusive] possession... The question what acts constitute a sufficient degree of exclusive physical control must depend on the circumstances, in particular the nature of the land and the manner in which land of that nature is commonly used or enjoyed … Everything must depend on the particular circumstances, but broadly, I think what must be shown as constituting factual possession is that the alleged possessor has been dealing with the land in question as an occupying owner might have been expected to deal with it and that no one else has done so."
and also:-
"Where the land was previously open ground, fencing is strong evidence of factual possession, but it is neither indispensable nor conclusive."