Lots of red flags OP. There would be no way I would buy the property without the land. Imagine buying your forever home with a party caravan next door - horrendous.
I think this sale is salvageable though. You are in a very strong position and if they have a good estate agent, they should be advised to sell the land.
*You are cash buyers
*You are flexible on moving
*They haven't had any other offers - how long has it been on the market?
*Although you found out about the caravan from a neighbour, it should have been discovered by a good solicitor in the searches, so any other future potential buyers will probably have had the same concerns.
I would carry on looking for property. They may need to confer with the rest of the family which is why it is taking time.
I think it is perfectly reasonable to set out a deadline on this but don't give them a really tight deadline - ask the EA how many households are involved in this sale - the more households, the longer it will probably take.
As previously stated, you are in a really strong position - you need to spell this out to the EA and they need to make this clear to the sellers, particularly if they have been trying to sell it for a while.
If this sale progresses, make sure you get a very good solicitor and also keep in touch with the farm next door.