I think the link to the article, I posted explains how this is not an uncommon thought for people new to home education. Most people reconsider when they think about what these freedoms mean, as you have done now.
And to be fair it wasn't a good day for me when you posted. I'd just heard about the latest bullying and underhand tactics of one LA local to me (but not the one where I live), against a particular family. It's almost beyond belief what they have done now 
Although not every LA behaves this badly-there are definately a number that do-including saying that it is unreasonable not to have home visits/show work......because they have so many other local families who apparently love meeting with them. I can't tell you how many times I have heard that said.
So you can imagine that I am not about to give a single inch on this one.
As ommward says, we came sickeningly close to loosing our freedoms and we are still here because we made a choice to make a stand (long enough for the last Government to run out of time!)
There is room for all types of home education. However you choose to home educate, with home visits or without, school at home or autonomous, we'd defend your right to do so.
But I think we've demonstrated our strength of feeling about doing something that will result in a loss of someone else's freedom to choose.