You're correct that the LA only has a legal duty to intervene if it appears to them that your children are not receiving a suitable education. That duty is reactive rather than proactive, so there isn't any legal basis for them to go round checking up on every home educated child in their area.
However, the political landscape has changed in recent years. Government is keen to introduce laws restricting home education. The Department for Education is actively egging LAs on to do routine inspections. Their guidance to LAs is non-statutory, but LAs do tend to follow it. That guidance advises them to make contact annually to request information about children's education, but also recommends that once the LA has established that the family has a good track record, there's no need to demand much detail.
So you are right: you have no obligation to respond to these enquiries. You can dig your heels in and wait to see whether the LA will follow through with their implied threat to issue you with a School Attendance Order and then prosecute you for noncompliance. If they do, you will need to present a defence to the court, which would take the form of evidence that your children ARE receiving a suitable education. If you get good legal advice and ensure that evidence is good, in all probability you would win.
But there is a small chance that you would get a quirky judge who might rule against you, so there is that risk. Also, legal proceedings are a hassle you might want to avoid. The pragmatic approach currently being recommended by the main home ed charities is to write an annual report outlining your children's education, but refuse to accept home visits, supply any "schoolwork", or give future plans. A good report should stop the LA.
In all honesty I don't know what I'd do if I were in your shoes. I am very opposed to allowing bureaucrats to engage in mission creep and try to push people around outside of their legal remit. Fifteen years ago, when my LA tried to demand annual home visits for my older child, I did quote the law to them and told them I would only engage with them if they demonstrated good reason to believe my child's education was unsuitable, and they backed down. But things have changed. What's more, my younger child is vulnerable and would suffer if forced to attend school, so I don't know that I would have risked it. Maybe I would have sent in a report. The LA never did become aware of her existence, and she's now 17, so I didn't have that decision to make.
Good luck with your decision.