I moved to one rural area from another equally rural area last year and was careful to try and 'fit in' here as soon as I arrived. I moved to this location specifically because it's a very cheap area and I have a little hobby farm and needed the land for my animals. I couldn't afford land anywhere else and have had to compromise on the size of the house so that I could afford the field too which is fine since I spend a lot of time outdoors.
What I hadn't realised though is that the farmer who sold me the field was previously letting someone else use it for their livestock (no idea if he charged them rent or not) and I think they are probably angry about me using it. Their animals were in the field when I arrived but they took them out without me asking. Since then they have complained about various things I've done, none of which have actually affected them, so AIBU to carry on using my field in the way I want?