It's just something people say, alongside 'money isn't everything '. However they absolutely do matter. They might not be #1 priority, but I'd be very surprised if anyone chose their partner without any consideration for how they looked. We're judged on looks all the time, rightly or wrongly, whether that be in relationships, job interviews and so on.