It's the same as decorating a room, say. You buy a roll of wallpaper that looks nice in the shop, but in that dark corner of your house it suddenly looks dreadful and garish. Or you buy a pot of boring looking neutral paint, and once it's on the wall it somehow looks amazing and actually makes you realise how nice your carpet is too (that you'd never particularly noticed since moving in).
A colour will either fight with its surroundings, or they will bring out the best in each other. Or something in between.
Have you never tried on a plain looking top and been amazed at how you somehow look prettier in it (because it flatters your colouring)? Or tried on a top that looked lovely on your best friend but when you look in the mirror, a pasty washed-out face looks back?
Lipstick, if you wear it? (As in, finding the right shade?)
I guess some people are more sensitive to this than others, but it's nothing mysterious. It makes a difference even if you don't think you consciously notice it. Personally, it gives me pleasure to find tones and hues that boost each other, whether they are worn by me or worn by my house. It's one of those small sensory pleasures that cost nothing. Maybe you are doing it without realising!
P.S. I feel I have waffled on so pretentiously that it would now be hilarious if I'm actually sitting here in clashing colours looking washed out and haggard, and you're shrugging at this thread baffled, meanwhile clad in an effortlessly flattering symphony of colours 😆