YANBU, OP. I have a similar neighbour at one of my holiday lets. He was nice as pie during the two years we spent renovating it (we were always clear from the start it was going to be let out and we knew he would probably find it strange having people next door again after the place had been empty for 40 years) and then the second lot of guests that stayed there decided to take an evening stroll down the field in front of the house. He had a go at them, saying it was private land and they weren't allowed in there. Unfortunately for him, one of them was a lawyer and politely quoted back the relevant part of the Land Reform (Scotland) Act 2003 which allowed them to be in there.
This did not go down well. The guests were mortified at upsetting him and left him a bottle of wine on his doorstep, which got taken back. I heard on the local grapevine that he was angry with me for not immediately going round there, so I tried to phone to arrange a time to go and speak to him, and was told he didn't want to speak to me or see me, so I sent a card apologising and saying I would put a large notice in the information book to ask guests not go into the field, and that got thrown back through the door at the changeover. He then rang the guy who rents that particular field off him and got him to stack wrapped silage bales in front of the living room window of my let. They're still there nearly a year later and now he's apparently saying he's going to put up a steel agricultural building when they're gone.
This has been a step too far for the lovely lady who does my changeovers, who lives further down the road - when she and her husband moved there, they really needed an agricultural shed for their livestock and he told them firmly that if they put in for it on that side of the road he would object, despite it not being in his line of sight from his house. So she has told him that if he goes ahead and puts in for one in front of my house, she will be writing a letter of complaint to the planning department, as will her husband (and probably quite a lot of other nearby houses, who are all fed up with his antics by now).
The really funny thing is that the one person he's trying to annoy (me) isn't remotely bothered by it, because I rarely need to go past it. The guests don't care, it's fully booked all summer and has just won a customers' choice award. The only people it's annoying are his wife (who is THOROUGHLY hacked off with him and gave me a really cheery wave and a hello the other day when I was there doing a bit of gardening) and the other neighbours in the row of houses who have to look at it every day. Apparently they're going to move into his mother-in-law's house when she dies, at which point hopefully the issue will go away, as we'd like to retire there when we don't have livestock any more (suspect our plan of buying that field off him, which used to belong to our house, is now not going to work out!!)