Weareeternal you can't really try to make someone give up this kind of acces when it's a terraced house as often the acces over the neighbours side is the only way they can get to the back of their house and the alternative would be for the neighhbiurs to bring his bins through the house. That would be very unreasonable.
I learned the hard way never ever to buy a house with shared access. Did it once when young and naive. The potential for disputes is immense, as we found out when neighbours son started bringing his motorbike up the ginned which WE were expected to maintain, and through the yard at the back of mine and two other houses.
Of course the motorbike owner and his mother, being in the end house, had walled off a bit just for themselves whilst insisting that the rest of the yard was communal... So me and the other two owners started puting our washing out in 'their' bit and the bike was soon kept in the front street.
The bad feeling it caused though, I moved the following year and will never ever even go and view a house with shared access ever again.
OP personally I think that the neighbour keeping is bin in his outdoor loo is ok, and he probably is ok to remove the bit of fence to get access to it as well. You need to keep things as amicable as poss when you live so close to people in terraced houses. I'd suggest to him knocking down both loos and building a shed type arrangement or bin store that you both use?
How many times WA week is he actually taking the bin through your property? You may find that you need to calm down about it all because he DOES have that legal access to bring his bins out through your yard.
Can another neighbour mediate for you, he may genuinely think that by using the old loo he has found a good solution. Does he know you are not happy about it?
I bet all these bins are causing all sorts of boter at my old house, we're have 2 bins, sacks for paper, and boxes for food waste all to go out on different days.