I am interested in this issue, though I am not very learned about it. I have not seen many unbiased sources that go into detail about what is necessary for slave holding to be without sin. I do know that the Church has approved of certain forms of slavery, and not just penal slavery, but other kinds as well, as long as the slaves are treated justly. But I do not know the conditions necessary for slave holding to be without sin.
In 1866, the Holy Office ruled that: "Slavery itself, considered as such in its essential nature, is not at all contrary to the natural and divine law, and there can be several just titles of slavery and these are referred to by approved theologians and commentators of the sacred canons.... It is not contrary to the natural and divine law for a slave to be sold, bought, exchanged or given."
At Vatican II, Gaudium and Spes no. 27 condemned slavery as "infamy" and a "supreme dishonor to the Creator." Since God has given laws regulating slavery and has approved of slavery under certain conditions in both testaments, this seems to be accusing God of approving "infamy" and a "supreme dishonor" to Himself, which is impossible and offends my pious ears.