Correct advice is in fact to tell the priest, and if the priest does nothing, that's on him. You are not in charge of the chapel or responsible for how people dress or behave. You are responsible only for yourself. If the priest does nothing, you can go to the priest's superior (if he has one). If it's an occasion of sin for you, then you should go somewhere else.