I think as has already been said that you should try and contact some other home educators in your area so that you see/feel more as though you have made a natural choice.
I can tell you as someone who home educated after her children had started school, that I fervently wish I had never sent them in the first place. In fact I envy tremendously those H.E. parents I meet who just never sent their children to school. I sent my children to nursery and school purely because that's what everyone else in society seemed to do. I skipped to the nursery when they started full-time, telling myself I would have lots more 'me' time. I never questioned why I should need this 'me' time, or if deep down I really even truly wanted this 'me' time. It was something I simply never questioned.
It is my opinion that you and your son are in an extremely enviable position; your son need never experience the notion of learning being different or apart from what he already does in life naturally, and you have huge scope to stimulate and follow his interests in areas and to depths he would never touch on in school and at whatever pace he wants to explore.
It sometimes happens that school can diminish a child's natural curiosity for learning because everything is separated into boxes: Maths-separate-from-history-separate-from-Art etc: Your son need never experience that.
Regarding L.A: You don't have to inform them, they have no right to demand to see you either, but it's up to you. A few home educators do find their LA representative useful and supportive, others say they find them unhelpful and not very understanding. We are known to our L.A. but have never been contacted. I really do think it is entirely a personal choice how you choose to go in this area, and you shouldn't necessarily assume they will be hostile. I do keep loose records (pictures, work, tickets from museums we've visited etc) just in case the need arises.
With regards to other people's opinions: It is natural, especially on the odd trying day or bad week to have a 'Am I doing the right thing?' wobble. It may seem then that everyone else has it easy, they drop off perfectly happy Johnny or Sarah to school and get on with life. I have known (actually I had this experience only last week Sunday in fact..lovely it was too...
) people who hardly factor in my life to get quite ridiculously upset about the choices I've made with my family. Sometimes I do get upset, other times it rolls off my shoulders - usually by the time we take our next visit to an empty museum or zoo at term time, or the next time my child finally masters some math topic -which he thought he was rubbish at in school- seemingly all at once and out of the blue.
I do think some sort of networking even if only on line is important, but aside from that don't worry about anything and enjoy everything! Just enjoy!