Dressing appropriately is a social code that we all follow (well, most of us do!) partly out of respect for one another and partly out of common sense.
Most people wouldn't turn up for work wearing a bikini and sarong (respect) or wear gardening overalls to a wedding (respect again) or an evening dress and tiara to the beach (common sense!).
There's absolutely nothing wrong with any of those outfits per se, but they are appropriate in particular circumstances and inappropriate in others.
To me, pyjamas, slippers, dressing gowns and the like are primarily for sleeping in and not an appropriate thing to wear when out and about.