I'd say it depends on the circuмstances and if wearing the hat is required. For example, some men in the military may have to wear a hat (in my country, a beret) while in uniform. If they wish to attend Mass in uniform or need to go to confession before training and they have no time to change, it becomes a necessity and can't be circuмvented.
I believe there is an exception for men in the military.
Dad told the story about how when he was in the Coast Guard Academy a friend who was also Catholic invited him home for dinner. It was a Friday. His friend told his mother that they had a dispensation from the rules of fast and abstinence so they could eat meat on Friday. His friend's mother replied, "Not in my house."