Trouble is that if they can't read they can't access most of the rest of the curriculum, beyond a certain age.
How long do you let it go on, attributing it to being a "late bloomer" rather than educational neglect? And by the time it's realised it's educational neglect, if it is, is there still time for the child to catch up with their peers?
At least with schoolchildren falling behind there's a few professionals who can give a realistic appraisal of what's happening (e.g. spotting dyslexia), and who have the skills to put in place remedial steps.
When genuinely illiterate gypsy and traveller parents are allowed to claim they're home educating their children from age 11, we have a problem, and those children are being failed by absolutely everyone.
A lot of the time it's not the kids in HE groups etc who are the issue. It's the families who claim their child is home educated, while providing no education whatsoever, only a religious education, or as a means to reduce the chances abuse will be picked up by a professional (as happened with Khyra Ishaq).