I should clarify that I didn't mean he'd threaten you with his guns.
Most farmers are pretty well enmeshed in their local communities, if you go up against him you'll probably find out the hard way that most villagers are wither his friends or get something from him on some way , so are likely to back him over you.
For instance in my village:
The village hall adjoins one of my fields and doesn't sit on a big plot so on special occasions they use my field for parking, bonfires. fireworks, camping etc.
I don't charge them for it and am happy that I can contribute to the local community.
Most villagers walk their dogs on my land, many I trust are allowed to let them off leads and I am not too worries about making them stick to footpaths.
A few of them fish my lakes - for free.
There's a couple of brother who have trails bikes. I allow them at certain times of the year to scramble their bikes on specific bits of the farm.
One family had a wedding last year, I let them put their marquee in my field behind their house for the reception and to nick a load of hay bales from the barn to use as seating. Again, I didn't charge them anything.
If a sticky break plonker goes up against me over something so minor or petty as this, all these people (most of the village) are siding with me.
Be careful about starting fueds in little villages. It will come back to bite you on the ass.