I'm a big fan of hats, and I wear them whenever I can get away with it. That said, yeah, the color is a problem. In my experience brimmed hats should be black, gray or another such subdued color.

If you want something bright, your best bet for getting away with it is a beret or a kangol-type thing like this.

And don't get me started on my secret love, pith helmets.