There is some impact and there probably are things that some schools could do better, but it is hugely complex. I'm not even sure the select committee found the answer to this one. Societal factors was probably a better choice of words than parents.
Poverty of aspiration is something that I think has been discussed here before. Linked to that is probably the value people see in education, particularly as a way of pulling people out of poverty. It's not that easy to educate children that are not in school because something more interesting has come up.
In many cultures there's an attitude that education is the job of parents and schools. So if a child is not doing well at school parents will find additional work to do outside of school to ensure they keep up. There's also a much less fixed attitude to attainment and more of an attitude that that what you get out is what you put in.
You also have to take into account the fact that, particularly in the case of first generation immigrants, they are probably a selected group of motivated and highly aspirational families. Which probably has an impact on the expectations they have for their children.
I also suspect that many children may be being assessed in Early Years in a language that isn't their 1st, that they may be in the early stages of speaking. The fact that white British children are the highest attaining at that age may be falsely reassuring.
You can probably manage most of these things to an extent in primary. It's probably secondary, when children begin to have more responsibility for their own attitudes and learning that it tends to go wrong.