I had this in my previous house.
The two bedroom houses (which mine was) all had access to the back and were semi-detached. All the three bed houses were terraces with no access.
It was a council house and due to some issues I had a lovely councillor who was helping me, and he turned up the day I moved in.
He noticed that there was a gate in the middle of my back garden to my neighbour's house (the three bed terrace). He said I should lock it immediately.
It's hard to explain. The two bed houses were set much closer to the road, so the gate to my back garden was level with the back of my house, but half the depth of my neighbour's house. I had a fence and gate there, the same as all the other two beds.
The back garden was lawn, but half-way down was a gate in the middle of the fence, to next door. This was locked on their side.
I contacted the council, who sent someone to talk to my neighbour, who said that this was their 'access', and left it like that.
Short time after, I was in the back garden, at night, when people came through their gate and walked through my garden.
It got worse when next door decided to build a conservatory. This meant that they had workmen, parking on my drive (the lorry used for concrete mixing actually cracked my drive).
Their workmen used to leave my gate open. Meaning that anyone could walk into my garden. Their gate, however, was locked.
I got fed up, so asked the council to check with their surveyor/land person. This confirmed they had no access. They sent a letter to both me and my neighbour, saying that there should be no gate. After this I padlocked my gate from the inside, but still allowed my neighbour to have things like turf, etc, brought through.