I know both can be thought of as colours, among other things such as the flower, but to me they have drastically different images. For Scarlet the phrase "a scarlet woman" comes to mind, and for Violet the phrase "a shrinking violet" comes to mind. Scarlet is a colour associated with danger and sexual passion, while violet is a colour which is associated with mourning, and it means modesty in flower language.
Scarlett also reminds me of Scarlett O'Hara in Gone With The Wind. She was beautiful and attractive, but also impetuous and selfish and very strong-willed. Not really a "good woman" sort of character.
So they have such drastically different images to me. I like Violet much more. I love blue violet flowers and violet perfume, and I think Violet sounds very sweet and demure, and I personally would prefer sweet and demure to feisty and the other connotations.
I think Violet is rising in popularity now, so though Scarlett has been more popular, I don't think Violet's a rare name either