Honestly, if you're wearing clothes that are ragged, they're going to also be faded and stained, even if they're clean. It's not about how you look or how attractive they make you, it's about maintaining a decent standard of neatness and self care. I find myself veering towards comfort over style too, but too far and it's bad for your MH. If your kids are mentioning it, then it's probably quite noticeable tbh, and I imagine you have their friends round in non covid times, other parents doing pick ups at the door etc.
I don't like bulky heavy clothes, so for me the answer is to layer up. I have bought some very soft, thin, almost jersey type hoodies and sweaters, long line so they cover my bum, and I wear them with high waist leggings and a vest top. I only wear sports or comfort bras, which helps. I also have a pair of slipper boots with a rigid sole, so although they're comfy, I can slope about the house with them and nip out to the bin or answer the door and they don't look like slippers as such. Hair clean, brushed and up in a loose clip. Tinted moisturiser every day, purely because it's SPF 30 and even if I don't leave the house I'll be in the garden,and I'm trying to protect my skin in advance.
Anything with holes, either mend it or chop into rags. There's no need for anyone these days to be wearing holey ripped clothing.
I do actually know someone who dresses like you describe, and it used to be that he'd only wear his ripped and stained old clothes around the house, but as time went by, he started nipping out to the shops, because who was going to see... Then he might as well nip to the other further shop while he's at it.... Then he might as well pop in and see x on the way back home....and now he wears the clothes all the time. He's lovely, but I know his children, and they find it embarrassing, because it's got to the point where it's just not appropriate