Gosh, in nearly 60 years of life I have never heard of this before. People in mid terraces, accessing other peoples gardens to get in and out of the back/take the bins out! As has been said, it was probably normal/OK when the homes were built, and many people probably lived next to friends/family members.
In the little cottage we live in, when we moved in, (just over a decade ago,) there was a little 3 ft gate at the back and a 3 ft fence, (on the one side,) so there was NO privacy at all from the one neighbour ... When we moved in it was a 50 y.o. single/divorced man, and every time I put the washing out or popped into the back garden for anything, he appeared. Started chatting shit/talking about himself/boring the shit out of me. 😂
I was really unnerved that the 3 ft gate was there and gave him access to the dining room window, the kitchen, and the back door! (Coz he could just walk straight into the garden.) Several times he just walked though the gate and came in and started chatting when I was doing the gardening or sitting there sunbathing!
So we got a solid 6 ft fence put up. NO GATE. He complained and said he 'enjoyed our chats.' Errrr, sorry mate but I didn't. I said we wanted more privacy, and security. He was like
Seemed genuinely confused that we didn't want the random man next door being able to walk into our back garden through a little gate, and peer at us over a 3 foot high fence! There was such a lack of privacy, that we only stuck it for the first summer before putting a big fence up! He never came out when my DH was out there funnily enough. 🤔
Fortunately he moved out a year after we moved in, and a lovely couple moved in who are still there and have 2 gorgeous little girls. 
Oh and @OneGreyBiscuit I would accept it if I were you, with it being social housing. You never know when the next one is going to come up, and as pps have said, you could have a decent neighbour here, and the next house could have a much worse one. And yeah, as some people have suggested, cordon off/fashion a walkway for the neighbours to walk through, and privatise like, 80% of your garden. Give them a pathway to walk across to take the bins out and get them back..