Maybe you could think of something along the lines of, 'Thank you for your opinion, I have thought long and hard and this is the choice we have made for the moment'
Sometimes people shut up, if they think the option to go to school is still there for later.
And while you say it-hold in your mind a lovely sunny day at HesFes with all of those well socialised children and young people, getting on with their activities, having fun whilst learning!
You know that his preconceived ideas are just not true-you have seen it first hand in large numbers.
Have you joined/met the local HE group yet. Having support and regular home ed meetings in your local area, will be some reassurance to yourself and possibly to him too.
Don't worry about him grassing you up to the LA -deal with that when it happens-and it might just be that your children will be reading by the age of 5.
And if they are not, well there enough of us about whose children didn't to give you support.
"but I can do better...or at least I thought I could, until he told me that I couldn't"
and I met you the day before he said that-and I am certain you can.
He can carry on saying what he likes, you can listen, nod, smile and think of HesFes and do what you really want to anyway.