I live on the end of a terrace with no access to neighbours gardens apart from through my side gate. My neighbours have a large wooden building in their garden that they put in a couple of years ago and took my fence down to get it in, the gate was never put back properly and now doesn't bolt properly. They have now put a note though my door today telling me they now wish to remove it and will need to take my fence down at 9am tomorrow, I'm not even going to be in! I responded saying that that's not convenient for me and I want to be in if they do it and that won't be until Friday. I would rather them not do it all tbh. Then about half an hour ago they were knocking loudly on my door for about five minutes and I ignored it. Aibu? I don't even think my fencing will withstand being taken down again.