The honor of God is at stake. Which you dont seem to care about
It's not for everyone to take it upon themselves to be a vigilante. It's not always prudent. It depends how immodest the dress is? and if the ushers refuse to do anything about it. I would stand in the back or move if it distracted me that much, and then contact the parish priest about it. I've heard that immodesty at Mass has become a problem in warmer climate parishes like Phoenix and Florida.
But think about if you tap that person on the shoulder, and they take offense and never come back. Is that charity? It's better to have a fellow lady talk to a lady to avoid the "he was looking at me". Perhaps the newcomer doesn't understand because they're new to tradition. So many have adopted the standards of the modern world and they don't see the problem in it. A gentle word is better. But anyway, it should be a job for the ushers. Maybe there should be a pamphlet on "modesty and decorum in God's house", and the ushers should hand her one. haha I knew a parish where they had a large suit jacket in the closet and the ushers would drop it over the lady's shoulders if they had to. Crank the AC in the summer and the ladies will bring coats.