What others spend is not really relevant to you, everyone has different budgets, requirements and priorities.
You need to look at what works for you. £120-130 pw is above average for your family size, but good food appropriate to your needs should be a high priority and unless you're paying off debt, or struggling to cover essentials and a reasonable amount of disposable income, there's no reason to be unnecessarily restrictive.
On the matter of dairy free, I'd concentrate on making naturally dairy free food as much as possible, rather than using substitutes, which are often expensive, unhealthy and not very nice.
You mention using a lot of oat milk. I think you can make this fairly easily, which will be a good saving as it's just oats soaked in water and then sieved. Set your OH on this task, as he's the one keen to save money.
Many men complain about how much their partners spend on food, but take no part in meal planning, budgeting, shopping or cooking.
Unless your partner is on board with the work of meal planning to a budget, happily eating cheaper food and foregoing snacks and extras like alcohol and soft drinks, brands, sauces and anything except basic core food items, he doesn't get to dictate the budget, especially as you allude to significant spending on gambling. Is he really expecting his family to cut down on food while he's gambling?