I know I keep popping back to this one but, one more thing!
You had a bad patch financially, your OH got stressed and lashed out at you. What is your assessment of the ongoing financial situation? What I'm getting at is, was its badness short term / part of a longer term problem / partly a question of perception? Do you agree that there is a financial problem to solve? Or is your view that by continuing to live frugally, you'll get by for years to come?
It seems to me that your DH has identified a problem (presumably with his income stream, since that sounds varied, whereas yours sounds steady) which he needs to work out how solve (unless you play a key part in running his business, which you haven't mentioned).
The answer may be that, as a family, you need to diversify your income sources, or that you need to swap the smallholding for a higher income job. But it might also be that there are changes that can be made to his working pattern.
What isn't fair, is for him to identify that there's a problem, then expect you to solve it - by doing something that bears no relation to the problem's immediate cause, which must be to do with his work.