I'm in that position. Maybe when someone speaks to me about it, I should tell the full story?
I stopped work due to bereavement, relocation and then illness, and also now, even though the illness is finally getting better, I have been out of the job market so long I would really struggle to get back in.
I am volunteering now, but I know I have a long road ahead of me. Until recently due to the state of the economy and jobs market in this country, I would have genuinely felt awful about taking a job that someone else might have needed a lot more than me (a single parent or a person whose partner was also out of work).
My career was in a sector with a severe skills shortage at the time, so taking a job someone else could have had was never an issue for me before, but having been out of work for 5 years now, I would struggle to get back into that and so would be looking for quite a generalist job now. In my area, the jobs market is getting a little bit better the last few months or so and places are struggling more to get staff, so I can see how maybe it wouldn't be a concern now.
So I do feel grateful that I haven't been forced to go back at the expense of my health and that I do have some time to rebuild my cv.
And some of the reason I have been able to do this financially was because my mum died and she left me some money which I could contribute to the household even though I wasn't working (I paid off a chunk of the mortgage and then we remortgaged to reduce our monthly repayments). I would much rather still have my mum and not have to look after her whilst she died a horrible, horrible painful death, not have had to resign because she needed more care than the system was able to give (especially as she was so terrified, anxious and in pain I felt she needed someone she knew and who loved her around for her last weeks).
Also losing my mum when she was relatively young (67) made me realise that just working away as hard as I could until I retired and then enjoy time with DH and DCs then maybe isn't always on the cards for everyone, so if you have the chance to take some time with the people you love, take it while you can sometimes, don't always put things off until tomorrow.
So maybe there are a multitude of more complex reasons behind a simple statement.