After spending many years in the U.S. Army, I've found that I just feel uncomfortable out of door without a hat.
Of course, I am also very fair skinned and if I don't wear a hat, the top of my head gets sunburned, so I wear a hat for practical reasons.
I used to wear a baseball cap with my favorite U.S. Army division symbol (my first assignment--the 3rd Infantry Division, which was, at the time, guarding the Fulda Gap in Germany). But I got very tired of people "thanking me for my service" after 9/11 and during all the wars since. Frankly, I was in the army during a period that was mostly "peace time" (or what we have been accustomed to think of as "peace".
So, now I wear a non-descript hat that is not a baseball cap. I wear a bowler when I wear my suit and tie. I also have a top hat for very formal occasions, but I don't find that I have many occasions to wear it and I don't think it looks as good on me (plus, I can't wear it in the car).