lymiemum I am wondering if the focus on the level he is reading at is a bit of a red herring.
I'd be interested to know how he is learning to read new words, so how sound is his phonic knowledge?
Or does he guess words or try to work them out from the pictures (just as the old style ORT books were designed to be used)?
If his phonic knowledge is really sound, and when he comes across a new word he sounds it out and blends it, and then after a few times of this learns to say it without blending then I am sure you are right, and his late talking just means that it will take him a little longer to catch up to his age appropriate level, but catch up he will.
If, however, his main strategy is to guess from pictures, or from the first letter of the word for example then I think you have a different problem that would require a different approach. In this instance ensuring he gains sound phonic knowledge will, imo, pay dividends in helping him to move on in his reading.
Regarding your view that he is surrounded by super reading children, the normal or usual range for Yr 2 would include gold, white and lime levels as well as free readers and yes these children are at a very different reading stage than a child on level 4, but, it is possible for a child (with the right support and when things click for them too) to literally start whizzing up the levels, going from red to white or let's say 8 or even more levels in a year. Children that are many levels below can and do catch up and even overtake children that were previously way ahead. Not all do however, which is why I think it's worth fully understanding whether it's a matter of his speech delay or not having all of the tools he needs to enable him to make good progress.
Super reading children's parents (of the bragging variety) are most definitely best ignored!