Notice that he adds "without just cause". "Just cause" can be rather light, such as that you simply have to interace with them given some situation you're licitly in. He also adds "elegantly" dressed, and I do suspect there may be a translation problem there, where he means more immodestly dressed, in general, or "attractively" (vs. "elegantly" in our modern sense of the term). So it's OK to look at a woman who's undressed, or dressed in an un-elegant (though extremely immodest) way? He also adds the qualifer "young people" ... to the exclusion of the elderly, and I suspect that he would also exclude obese (or ugly) people, though obesity was likely not common in his day.
So, putting together the various qualifiers, he says that it's venial sin to look at attractive women without just cause. As with a lot of matters dealing with moral theology, the degree of attractiveness depends not only on who you're looking at but who's looking. Some individuals might have a much higher threshold for what they find attractive than others, sometimes due to their degree of virtue. I recall the story of some early Christian saint who was not paying attention and walked by mistake into a woman's bath-house and didn't even notice, since his mind was so absordbed with God that he barely noticed his surroundings. Others, who are of a higher degree of virtue, find God so beautiful that women are ugly by comparison and don't interest them. But even for such as these, there might be the entirely-involuntary pull of the flesh, due to one of the effects of Original Sin, concupiscence.
St. Alphonsus can therefore be summed up as saying that unnecessarily looking at women that YOU happen to find attractive would be a venial sin. Put another way, if you look at a woman who's attractive to you for NO REASON WHATSOEVER, that would be a venial sin, since the only remaining reason would be precisely because you find her attractive and are indulging in said attractiveness. But if you happen to catch sight of some, here or there, as you go about your legitimate daily business, or even, depending on the situation, are required to interract with some, those would be justifiable. But if you just look at them for no reason whatsoever, or even without sufficient reason, as he puts it, that could be a venial sin.