I really don't think it's an issue of whether you have anything to hide. I wouldn't be keen to allow the police to come searching my house and every other house in my street "just to check" that there were no stolen goods inside, when there was no evidence to suggest I'd been involved in a crime. I am sure they wouldn't find anything. But that isn't the point.
In my case, I think I might possibly have considered accepting a visit if my LA had been completely above-board and polite about it. But they think it is appropriate to mislead the public and imply that people must be inspected at home annually. They even specifically told me on the phone that there was a legal requirement for this. And I know they know better, because I and other local HE parents have written to them repeatedly over the years to "remind" them of the law! Frankly, this behaviour infuriates me and doesn't give me reason to trust them any farther than I can throw them.
As morethan says, you are quite entitled to decline your LA's pushy "offer" of a home visit. The main thing to do to avoid any hassle is to ensure you always reply to any communications from them. So, if you don't want a visit, write and tell them so. You could offer to supply information in writing instead, if you are happy to do so. There is not even any legal basis for them to request information in writing, however. Unless they have some reason to believe your child is not being educated appropriately, they have no statutory duty to be involved with your family at all.
As far as I can tell, most HE families prefer to supply some information to the LA even though they don't have to. Then they are in a position to point to it and say, "Look, I have already given you more than the law requires. Unless you have specific concerns, there is no need for anything more."
Once you have decided how you want to play it, if you need any help with phrasing a letter, just shout! Many people here will have been in exactly the same situation as you and can help you quote the law if you want.