You are wrong.
When God asks you why you had this or that bad thought, do you think He will accept "but she dressed provocatively "? No.
Free will demands that we control ourselves in all things.
It most certainly makes a difference!
We're talking about casual glances here, not leering. If the breasts aren't exposed for example, men don't have to be confronted with the choice "consent or do not consent", "enjoy or do not enjoy". Once the breasts are exposed for all to see, then each and every man within visual range must test his virtue.
Women are not guiltless when they cause men to have to test their virtue. You can pass the buck all you want "that's his problem".
God will judge. Try to quiet your conscience all you want.