I've taught children who have English as their second (or third) language and absolutely agree that it is the vocab that they often struggle with, particularly later on. It's easier to perceive a child as having a really good level of English if they are in Reception/KS1, but as they get older they are expected (as are the native English speakers) to have a much wider vocabulary and comprehension of the subtleties of the language.
For example, a five year old may describe something as 'wet'. An older child would be expected to understand and use more subtle variations of this, such as 'damp', 'moist', 'soaked', 'sodden', 'slick', 'humid', 'clammy', 'watery', 'dewy', 'saturated, 'hydrated', 'dank', 'aqueous' - you get the idea. They could say 'red' - or they could say 'maroon', 'burgundy', scarlet', 'vermilion', 'crimson' etc. A native English speaker will have been exposed to far more of these terms, and will be far more likely to understand the subtle differences between them and in which context they should be used. Their reading comprehension will be lower due to this, which will then affect almost all of the subjects they are studying.
DS2 is in Y1 and 40% of his class are ESL. This fits with the general percentage in his school overall. However, the school achieves excellent results and generally scores SATS results which are top or second in the area. I certainly don't feel that DS is missing out educationally because of the high number of ESL pupils in his class 
I imagine a lot of that is due to the school having very practised methods for teaching ESL children - they are very experienced and spend a lot more money on this area than most of the primary schools that I've taught in. Also, many of the parents are already fluent English speakers who work at the local hospital or universities - they probably have a higher level of academic English than many native speakers. They often, though not always, have the money to pay for additional tuition outside of school. They ensure that their DC have English books at home, watch English TV and read English newspapers and make sure their children are enrolled in lots of groups/activities with native speaking children. The children who don't speak any English except when at school (maybe their parents don't speak fluently, they can't afford tutors, they only socialise within their own communities and watch TV in their first language) find it noticeably harder as they get older, ime. They often end up not speaking either language to a high academic level.
I was thinking about this only the other day actually, because DS had his friend over for tea. His friend started Reception mid year, so about 18m ago) and he spoke no English at all. Listening to him chattering away to DS, I was marvelling at how he's become socially fluent. He can chat away about school, games they were playing - you'd not necessarily know he couldn't speak a word only last year. But he is far behind DS in his reading, maths, everything really, despite being obviously bright. His brain has had to learn the language first, so it has missed out on picking other things up. Still incredible how fast children can pick up so much!