I haven't worked outside the home for many years.
Not once have I felt the lack of it in terms of brain stimulation- but I have many hobbies and interests beyond baking and crafting (although I do love crafting, to be fair.)
If you're more independently-minded there are a million things you can turn your mind to- learn a language, learn to sew/paint/plumb/decorate/lay carpet/garden etc, do a degree, volunteer, pursue a goal (marathon/triathlon e.g), read (good books, trashy books), improve your knowledge, study the stock market, start your own business, take up a sport/martial art...
I've always hated the kind of lazy thinking that inspires people to believe only a workplace can provide any kind of mental stimulation. If you need direction or help in that area, then of course having a job will be great for you.
It's also very silly to think that you're not contributing to society if you don't do paid work. Looking after grandchildren-to give their parents a break, or to provide childcare so that they can go to work themselves- is being useful. Volunteering is being useful. Being useful to the larger world around you is not limited to doing paid work.
Financial independence- that's different. Having a job= money of your own. That's a good thing.