I think doing very little can breed listlessness and brain fog and the less you do the smaller your world can become etc
I totally agree with this. I know someone will tip up to say that we all work too hard and should have more time to "think" and "be" and "potter" etc, and that's all good and well. There's definitely a place for building more down time into people's lives and not having work define you.
But I've seen so many people basically atrophy without purpose in their lives and for most of them at the root of it was not knowing what they wanted to do as a job. My mum is a prime example. She gave up a hugely rewarding career to have children, basically never went back to work (this was the early 70s) never really got her mojo back and bitterly regretted it. And her world just shrunk. She didn't really have any hobbies, she lost most of her friends and spent most of her later years desperately searching for purpose. She was desperately bored by housework etc but never had the oomph to direct herself elsewhere.
There are lots of ways to have a rewarding life outside of paid work as plenty of people have pointed out, volunteering being a prime example. But the reality is without some experience of the discipline of paid work and the sense of purpose that having work gives you and the experience of working to deadlines and working alongside other people, its harder. There are probably equivalent ways of getting this discipline: running church events or guiding events or something. But for the vast majority of people the most logical and rewarding way to do this is work.
Working alongside other people teaches you all sorts of things which you carry with you through life.
People who are out of work for long periods of time without much else to do tend to lose a lot of confidence, they lose networks and without a lot of discipline they lose structure. This isn't a foregone conclusion and with discipline you absolutely can instil this in your life. But its much much harder if you haven't worked.