OK - you have been off 4 days with a chest infection/flu illness, so quite physically ill and unwell. You are now running low on food and also, while starting to recover, are going stir crazy from no fresh air in days. You go to the local supermarket to get food, and also to get a sandwich and coffee made by someone else to save on effort and also to give you a breather after the effort of driving there before doing the shopping and going home. But you are not allowed to have the sandwich and coffee while out, you must instead buy the bread, fillings, milk etc and go home to make it yourself. After which point, in either case, you are likely to collapse back onto the sofa/bed after wearing yourself out doing that shopping run.
Or you have been signed off (even in an office job) with a broken ankle. GP may allow you back in 2/3 weeks once it stabilizes and starts to heal, but you have steps to manage and a commute on public transport so GP says stay out 4 weeks. Does that mean that you shouldn't be going out for a coffee locally while you are off, and are resting lots but even less likely than the 1st example to need copious amounts of sleep (certainly once the initial shock has worn off and pain subsided and healing has started)?
I have an office based job. A pretty stressful one. Long days, commute by train, I organize pretty much all of our home life and managing DD. There are occasions where I have been off, actually sick, but feel guilty about staying off longer than the absolute minimum - but then go back in and need a further period of time off as I am not yet recovered. Yes there are some people who take the micky on sick leave. But there are many many people who don't, have a sensible attitude towards it (only taking it when needed, trying to head off illnesses at the first sight of them, building up their reserves when possible to cope with bugs etc). And yet need some time off - which may involve further recuperation before returning to work.
Last autumn, I was signed off work for a week, I went out on day 5 to get groceries and had to turn around and come home again as I was so weak. Yet I had managed to have a coffee the previous morning after dropping DD to school (because a long slow coffee and breakfast out meant I could fall into bed when I got home and sleep for most of the day having had food and liquids).
Going out for coffee doesn't always mean meeting 4 girlfriends and having a wonderful gossipy catchup for a couple of hours - it may mean having someone else make you a meal that you can sit and eat before going back home to continue recovering. How the HELL is that a bad thing and how is it saying that you are well enough to go to work? I can have a coffee with my mind in a whirl, completely unfit for actually working but looking reasonably normal apart from being extra bundled-up and flushed. My normal pace is 90 miles an hour - but those kinds of coffees tend to be at 10 miles an hour - and my boss is far happier to see me take the extra day or 2 to actually recover and be productive once I DO return, than to come in once I feel vaguely human and get almost no real work out of me for another week or more.