My kids are 17 and 10 now. It has been great.
It interests me to compare the experiences of my older "trailblazer" child with the younger one who followed after. The older one was quite conscious of "being different" because so many friends, family and strangers expressed surprise that she wasn't going to school. Some were openly critical, which worried her.
I've heard similar things from other HE parents. It can unsettle kids and make them wonder whether their family is doing things wrong. Adults all around will be "talking up" school in a well-meaning attempt to persuade possibly-reluctant children to go off cheerfully to school. Toddler books never feature children who dislike school for more than a day or two before they realise their fears were unfounded and it's all lovely fun all day long! Many of the kids around will be excited about school. Yours may feel left out.
Based on that, my main advice to you would be to meet up with other home educating families with young children IRL so your son won't feel alone in what he is doing. You may have to do some networking to find them, but even one or two families could be enough. If there really aren't any nearby, make the effort to travel to a group even if it's difficult and you can only manage it once a month. Alternatively, the home ed camping season is nearly upon us and there are many to choose from. That way, your son will know a few home ed kids, feel that it is a normal thing to do, and perhaps be excited about his future.
My younger child was spared all that angst because she grew up in a home ed community. By the time she was a toddler, we were well into home education. Her older sibling had loads of HE friends round all the time, and we went to many HE activities. School wasn't really something she thought about. Even now, she doesn't completely grasp that her education puts her in a small minority group. This is her normal. Sometimes I wish I could go back in time and give that sense of certainty and normality to my firstborn.