I wear one almost every day at some point. I ride a bike, and am far more likely to be judged (Iguess) for not wearing a helmet (same when I ride my own horse, although not others').
Years and years ago, someone told me I look about 18 when I wore them, so that was nice..... Also, for a brief time I was also told I resembled Kelly McGillis in Top Gun 
I don't now though (if I ever did), and I don't resemble the present KM either....
Wearing a baseball cap has the advantage that I am far more likley to go for a ride, or use the bike as transport in inclement weather, rather than find a tree to hide under when it stars to rain or not use the bike at all; although I might still do that if feeling like a wimp. Or a troll, if alongside the canal and taking refuge under a bridge 
What I would say though is, looking at the peak of the caps (I rotate several black ones, always the NY ones from Sports Direct, although once upon a time I did have an actual NY Yankees one as a gift from NYC - it was pink and I loved it), there is quite some fading - which would have otherwise happened (or damage) to my skin. I have literally no sun spots or freckles after years of daily riding bikes and horses.
The same with my hands as I usually wear gloves (sometimes without full fingers), and always for horse-related tasks. I also have only one freckle or sun--spot on each hand, which as a fair person is not many at all...... I was watching a Jane Fonda What's in My Bag recently, and she brought out some fingerless gloves that she wears for driving to protect her hands from the Califronia sun - so I feel endorsed by that 
I cannot imagine wearing a floppy brimmed sun-hat, delicate lace gloves and shading myself with a parasol for the things I do 