90 percent of brain growth happens before the age of 5. So I'm guessing most of those children will never catch up or be able to reach their true potential.
Whilst this may be true, and the early years are incredibly important, it is entirely possible for them and adults to catch up academically. With the right determination, facilitators who understand what might pique a desire to learn, scaffold that learning effectively and develop determination and mindset, it is possible. (Adam Grant's book 'hidden potential' is a great description of the hows and whys.)
What has more of an impact on not doing so are the things that hinder social interaction, communication and self regulation skills. (Neglect, trauma etc) The second great brain growth stage, puberty, can offer opportunities to further develop all this, again with the right support, but it tends to be harder. Not least as that right support is so hard to find.