I have been a home educator - for the reasons given upthread about bullying / anxiety - and am now a teacher (and have taught in a school with a very high percentage of Traveller children, in which group all the girls were removed from school at 11/12 for home 'education' / preparation for becoming a wife and mother).
So I can see this from all sides. i have never seen 'fear' as a response to HE, though I do know that my decision to HE my DS did cause quite a lot of other families to look very closely at the school we left, with the result that a significant number left to go elsewhere. I have seen a certain amount of antagonism / defensiveness in all directions - ie from HE families to schooled families, and vice versa - but i would say that this is in no way greater than that between any groups who have chosen different educational paths for their children - e.g. private vs state or selective vs non-selective.
Home education is not wholly good, or wholly bad. It's a bit like the 'are state or private schools better' debate - for an individual child, a particular state school may be better than a particular private school, and vice versa. However, home education (or rather 'education organised wholly by parents', because the term can include groups / classes / tutors) is very highly variable, even more so than the school sector. The private school sector can range from Steiner to Eton; the state sector from Tiffin to tough secondaries in highly deprived neighbourhoods, via Jewish schools and tiny schools in remote areas, but the variability in specific parent educator / child pupil pairs can be VAST.
The 'good' end of this range - looked at in terms of 'progress made by the child in all aspects of their life' can be exceptionally good. DS;'s first school rendered him a bruised selective mute with suspected autism; a few months of HE transformed him back into a bright, curious, courageous child once again (though the ASD traits, which turned out to be linked to acute anxiety, remained clear for at least another 5 years). However, the 'bad' end - again looked at in terms of progress in all aspects of their life - can be horrendous. Even the 'not very good end' - children with no SEN who are loved and physically cared for but not offered any form of specific education - can further marginalise already marginalised children.
It does HE no service to deny that the 'bad' end exists alongside the 'good' end. If a form of sympathetic oversight could be developed, that takes into account the fact that the decision to HE can be taken by those with greatest distrust of the authorities, either through being part of a marginalised cultural group or through experience in a school or school system that has caused actual harm, then this would seem to me to be a good thing. It would enable the good home educators (not those who 'produce the best clutch of certificates, but those who genuinely enable their children to make progress in all aspects of their lives) to be able to trumpet their success while removing the 'dark shadow' of potential neglect, or at worst abuse, which currently taints the sector.
It's a bit like having a boarding school sector that still includes Dotheboys Hall - the bad taints the good. However modern oversight has removed the worst end of the boarding sector, so the overall reputation and image of all those schools that still remain is improved.