You have my deepest sympathy. Is this a new thing or has he always had a hat fetish?
My DH loves baseball caps and collects them - he has done since being a teenager, wore them all the time when we first met as students but he is now in his 30's and it shows no signs of abating. Some of them he looks nice in, and as his hairline has receded somewhat, he does like them.
If he were a Ken doll and mine to dress, they aren't what I would put him in, but they are his 'thing', and to be honest, he wants for so little in life (he hardly drinks, he isn't into xboxes and suchlike, not fussed about having an expensive car or clothes), that I feel unreasonable making a fuss over it, so I have let it slide. He never tries to dictate what I wear, so it's only fair. Infact, I suggested that if he wanted one for our wedding (reception only!), I was happy for him to wear one - he was surprised and made up to do so. He considers that he will stop wearing them when he's an old man and can legitimately wear a flat cap.
YANBU, but I don't know what to advise you!