Are there any 'other allotment holders' though? The use of the word allotment has confused the discussion here from the start, because that sounds like a series of plots rented from another party. A renter might have sympathy for another renter.
The plot in question is not rented it is owned by the OP - is the same true of the others?
If what is there is a set of vegetable gardens (or just gardens - is there any restriction on what you can grow?) attached in law to specific houses, though detached by position from them, then what you have is 'other land owners'. Who are going to view a squatter claiming rights in a very different way.