Handsome men are more handsone with hair than without. Sounds like your grandfather had a good face.
Men have a proclivity to see in ourselves only what we find attractive, have you noticed this? I had my hair long for a short while, all throughout winter, which I may continue seasonally for function.
I cut it, women really like my short hair, I did not immediately like it myself.
If you go bald, who knows what other people will think. If I went bald perhaps women would like it even better. Hence I will not go bald. Here are photos below.
Shave it and go out into the world. I get very thick sideburns too, shave them. Others may like what they see more than you do.