OK this is the best I can do, and this is just one of many different formats I could illustrate.. Imagine it like this, a row of adjoining terrace houses. The front of each house opens up directly onto a pavement (not always the case) so they cannot keep the bins out front. They have to keep them in the back gardens but there is no back access from the gardens. Lets pretend for instance, a railway line runs down the back of the houses. So Terrace 1 can have a side gate on their back garden so they can bring their bins out. Terrace two cannot because there is no direct access to their back garden (houses both sides and its not reasonable to expect them to drag their bins through the house. So when the houses are built, an easement is created saying that Terrace two has the right to take their bins out (for example through Terrace 1's back garden) but they must stay on a specific path. That path is part of Terrace 1's garden.
The easement exists only because Terrace 2 has no independent way to reach its back garden. So the builder grants a limited right of access across Terrace 1’s land so Terrace 2 can do essential things like take bins out, bring garden waste through, access the rear for maintenance, deal with emergencies. This is quite typical for where I live in the North.
The bog standard easements don't give Terrace 2 the right to treat Terrace 1’s garden as a normal walkway, entrance, or thoroughfare. It also doesn’t give rights to visitors, couriers, tradespeople, dog walkers, cleaners, boyfriends, randoms. Unless the wording explicitly includes “agents, invitees, visitors” (very rare in residential easements), the right normally applies only to the occupiers. But the devil is in the detail and that's why the OP needs to check the exact wording.
Most easements of this type specify in some manner the exact route, the obligation to cause minimal disturbance, the requirement to keep gates shut, the requirement to maintain security, the obligation to act reasonably. They’re often worded like: “A right of access on foot only over the path coloured yellow for the purpose of removing refuse bins and maintaining the rear garden.” That’s the level of restriction you typically see. Not that I am an expert, but I come across this in my career on a regular basis. I hope it helps.
But some people have mentioned it can become prescriptive i.e. roughly speaking (this isn't exactly it) if Terrace 2 has been allowed to use that pathway in an additional manner, say as the main access to their house for residents, visitors and deliveries etc for 20 years then it can become what is called prescriptive and they in effect become entitled to use it that way.
From what the OP has said, I don't think this is applicable here (assuming its England) but we won't know until the land registry is checked.
But I am thinking if the deeds already grant a limited right of access, you cannot then claim a broader right (e.g., “I want to use it as my main entrance”) by prescription. The law says: you can’t prescribe against an express grant.
They already have a right of access. They cannot “grow” it into a right of way or a right to use your garden as their front door.
Their use is not “as of right”, they’ve been told off by previous owners, told off by the OP, apologising, arguing, claiming it’s a “public right of way” (which is nonsense). That is not peaceful, uncontested use. It’s use that has been challenged, which breaks the prescriptive clock.