Fence belongs to him and it is the kind with concrete posts and wooden panels that slot down in between the posts. We don't talk to each other. But every few months, without warning he takes the panels off completely exposing what we do in our garden like having the washing out or kids playing etc. We feel like we have no privacy or respect from him. We would like to somehow ensure he can't take panels off but don't want to damage his fence and don't want to do anything that is not legal. Are there non intrusive ways to keep the panels in place?