That's your main issue. Cost.
But we have to consider what we are paying for. We pay (paid) land owners for land management through the EU. So it's a hidden cost of farming we aren't seeing when we buy food directly. (And this is where Thatcher and the uk rebate was such a good deal...) This is going to change. There are arguable points in this area about how the EU hasn't necessarily been good for UK agriculture (yes I did say that), but this is where we need time to adjust to a new system and to have a robust plan for this.
We've got used to cheap mass produced veg and huge amounts of wastage, both at source and in our homes - something that certain celebrity chefs have made a song and dance about (including to the EU when it comes to fish). It is one of those things which is unsustainable in the long term regardless of our EU membership whether we want to admit it or not for bigger reasons.
In the short term we might have huge issues of our farming being undercut which could completely destroy the sector and make us extremely vulnerable in terms of food security.
But this is an issue which isn't going away either.
Longer term we might have bigger issues because of that food security issue, population growth around the world and climate change.
I don't know how you get through the intervening period whilst protecting UK agriculture (without potentially damaging and risky protectionism) but it is something we are going to have to give a huge amount if thought to. Axing farming and food standards isn't necessarily going to help that. (Though it will focus the minds of the better off seeking food which they trust the quality of - and maybe drive demand for high quality British grown produce. Whether there is enough demand is another matter).
In a similar vein our local farm has done better than some farms locally in recent years when there's been weather problems because its more like a small holding with a range of crops rather than having risk stacked up in a a single crop. It's also asked for investment from its customers in innovative ways so it could avoid going to the bank (and paying interest in that way).
So I do think there are things that can be done but it's certainly not going to be easy.
For us we took the conscious decision when we started buying from them that it wasn't just about the food that we were buying. It was a lot of other aspects to it - particularly the community aspect. It's certainly paid off for us as we've had greater food security and convenience during lockdown and of course there us still Brexit cliff edges looming.
Our investment (and that's how we've viewed it for a long time) is starting to pay off for the farm. They've expanded hugely and they started several community related projects as part of the business. And there's the employment factor - which for local teenagers unable to do anything else this summer has been brilliant.
The money staying in our local community rather than being sent to Eastern Europe makes a difference.
This is one if the few areas I perhaps start to sound more Brexity, but I do think there are major structural problems with our agricultural industry fir a large number of reasons which Brexit might make us face up to. Covid-19 certainly has exposed some if these issues.
And yes food insecurity and just general poverty play a big part in this huge problem.