Once they're out of infant carriers, they get a lot less portable for lift sharing. Toddler car seats are very awkward to move around.
Children's activities such as soft play or petting farms are often difficult to access by public transport in industrial or rural places. Older children's activities may not work well with public transport timetables.
Dropping off at nursery/ school/ work may not be viable without the flexibility of driving.
More food shopping to transport.
Children take up a lot of time. My non-driving friend could bus to me when her oldest was still using the pram. Much harder with two, and nigh on impossible to fit into a school day when the oldest went to school. For years, I couldn't pick her up as 2 adults and 4 children do not fit in a standard car, and although I now have a 7 seater, she is often a significant diversion away.
If you're in an urban area with a lot on your doorstep and great, frequent, reliable public transport connections (city centres or small towns tend to be best) it is doable, but for the majority of people in suburbs or rural areas, not driving would be very compromising on lifestyle and opportunities.