Yes, you're right Milly. But then again, for almost the entire time people have existed, they haven't needed to exist in a literate society. Children now, do.
A study comparing literate and illiterate adults from a similar background showed that the illiterates had great difficulty understanding that a word can be split into its component phonemes (eg., they didn't know what you meant if you asked them to say 'break' without the 'br'). Your whole understanding of words changes if you can conceptualize them as symbols on a page. This impacts on things like understanding that nonsense-words can be made up, understanding how to sound out words in another language.
I think listening to a story and understanding that it is being read out from visual symbols is quite important for developing a sense of how language can be structured by humans. It makes us more reflective. I don't know if it matters at what age you are read to, or learn to read, but with other (granted, more 'natural') processes like learning a language, the time at which you learn the language is very important. Take bilingualism: an 'early' bilingual who learns both languages as a baby or toddler, will actually use different parts of his/her brain to do so, than a 'late' bilingual.
Sorry if this sounds quite speculative, I'm trying not to overstate my case (which is easy to do): I am pretty thoroughly convinced that reading to children is important, but my research has to do with literacy and illiteracy in the past, and I know how difficult it is to 'prove' anything about the way the brain develops.