I know it's difficult to credit, but the low educational attainments of some home educating parents tend not to have an adverse effect on outcomes for their children. Compared with school, it's a different model of education, in which the parent is not the primary source of information but instead a facilitator. So if your sister is prepared to support her son, that is likely to be enough.
There is little good research on the subject, but an interesting British study looked at the educational attainment of children from various backgrounds who were school-educated and home educated. As is well known, parents' background does matter. Unsurprisingly, the children whose parents had completed the most education had the highest educational attainment, regardless of where the child was educated. In general, the study found that home educated children outperformed their school-educated peers who came from similar backgrounds. I'm sure it will surprise you to learn that the difference was greatest among children whose parents had the lowest levels of formal education. In fact, the home ed children whose parents had completed the least formal education were on average about two years ahead of school-educated children whose parents had a similarly low level of education. One might conclude that the best way a mum who can barely read and write could boost her child's chances is to take him out of school.
For what it's worth, anecdotal evidence supports this. I know a few HE parents who really struggle with basic reading and maths. But their children are average. It's because they aren't being "taught" these subjects by their parents, but are learning from everything around them.
I'd agree that it may be worrying if your nephew is stuck inside with his mum all the time and unable to get out. Will this be the case, do you think? Is she pretty much completely housebound and doesn't go out even to do shopping etc? Home education in itself doesn't confine kids to the house; it's really a misnomer in that respect. If you think his friends may be a bad influence then the best thing you can do is see if you can encourage him and his mum to meet some other people.
I don't think it's necessarily the case that he has been abandoned by the system. Maybe he and his mother have decided that the system isn't working for him, and they are abandoning it. But it's still there for them if they need it. If they are right, then that is a sensible decision to take. As the school system only has hold of kids until they are 16 or so, there is a real feeling within that system that time is of the essence, that kids must achieve before they are 16 because it will be too late thereafter. Home educating families generally develop a longer view: we witter on about education being a journey and not a race and we really do believe this. There's college and adult education later on if your nephew doesn't learn everything he needs right now. It will never be too late for him to get an education... unless he has such a miserable experience of education when young that it puts him off for life. Or, perhaps, if he spends all his time with people who think education is a horrible ordeal instead of with people who are enthused by it. Again, if you can encourage the family to meet other HE families in the area, this will help.
Probably this won't put your mind to rest much. But do send them our way if you possibly can. If their horizons are as limited as you believe, this will help. Even if you are mistaken and they are doing just fine, they may well enjoy being part of a supportive home ed community.