I know exactly what you're talking about and I'm sure most men do. Whenever I'm extra-scrupulous in trying to practice custody of the eyes, it makes the pressure worse. Because if you are going to block out all tantalizing sights, you would indeed have to have your eyes fixed firmly on the ground.
God's solution for me was to provide me with the assistance of St. Stanislaus Kostka, who I pray to most days for purity of mind. So while I look say 50% of the time, it has less effect on me than before.
Once you get some momentum you can resist the urge to look more and more. But from what I have experienced, you can't force it, if you are saying "Mary, help me" over and over, it's trying too hard, no one can live like that. You can't literally call on her at every second of the day. If you have Mary in your heart, however, then you are calling her all the time. That is why when you are at a high spiritual level, like certain saints, you really can avoid all temptation even to look.
On the feast day of St. Dominic Savio, my priest's namesake and patron saint, he spoke in the sermon about how St. Dominic practiced custody of the eyes perfectly, but he had a natural curiosity, so that in the beginning he would get headaches from the effort.
You're not going to go to hell for briefly looking at a girl in a skirt above the knees; but the more you look, the more you will have women on the brain.
It seems to me that with EVERY virtue, what it all comes down to is love of God. The more you love God, you are just filled with every virtue. So that is how to attack the root of all problems; advance in the spiritual life, love God more, live for Him more and more, and the old man falls away gradually. If you are trying to become virtuous out of pride, if you are patting yourself on the back for being so strong and resisting temptation, you will find yourself slipping. The more you are aware of your weakness, and the more you entrust yourself to the care of Jesus and Mary, the sharper your consciousness is of your own sinfulness and unworthiness -- which everyone says they have but VERY FEW really have, this is spiritual perfection -- then the easier all of this gets. Find more ways to work for God at every second of the day, become filled with Him more and more, this is the goal.
I have gotten my big toe wet when it comes to humility, I am on the path, though far from perfection; but this little taste I've had makes me want more. I'm seeing now how immersed I was in thoughts of the flesh even when I was baptized and wasn't committing mortal sins. I had to fight more, now it's easier, the curiosity is lighter. But I didn't really do anything to get here, just kept trying to get closer to God and seek His ways.