I grew up in the 1960s, so children who were non-verbal would have been filtered out and sent to specialised institutions. There were 'very shy' children at school (including me) who hardly spoke to anyone in the school environment. However, there was a strong pressure to conform, and we were labelled as 'shy', 'awkward' and 'sullen'. I spent a great deal of my childhood, wishing I could just go up and talk to other children, and feeling that I was a 'failure' because I could not.
I had some issues as a teen in the 70s and went to an adolescent unit where there were selectively mute children, but still I had not met any children who were non-speaking through autism or learning disability. I suspect children who could not speak by this age (except for those who had physical impairments that made it impossible to speak) would be in institutions.
I do not know much about the 80s, because I was catching up in life. However, my niece did not speak until she was 4.
My own children were born in the late 90s early 2000s. My eldest son's speech delay was identified by the health visitor who had worked with my niece. My mother in law then told me that my husband (her son) did not speak until he was four. She was a teacher and was scared that if he did not start speaking before he started school he would be considered 'ineducable' (an outdated term, but one based on an Education Act that had serious consequences for children with serious SEN before 1970).
So, at a guess, I would say there were a considerable number of non-verbal children in previous decades, but societal norms, legislation and the older predisposition towards institutional or residential provision would mean it would be unlikely that we would meet non-speaking people in our everyday lives, unless we had a family member who was non-speaking or we accessed or worked in the specialised contexts.