Lad,
It is clear, both by Scripture, the Council of Trent, and indeed Augustine's own words, that all men are "condemned" by Adam's sin, and that this "condemnation" falls upon infants who die unbaptized, who endure the "penalty" or punishment of that condemnation, which comes by way of God's "judgment" (again, Rom. 5:16,18).
No, it's not clear from Sacred Scripture. There's a completely different trajectory that the Eastern / Greek Fathers took regarding the distinction between nature and grace vs. in the West, where they followed the thinking of St. Augustine. St. Gregory of nαzιanzen, called "the Theologian", in his famous rejection of BoD, articulated the notion that there are some who are not bad enough to be punished but not good enough to be glorified. Again, I encourage you to read the CE article on Limbo. St. Thomas Aquinas clearly articulated the Eastern perspective, that elevation to the supernatural state is a free gift that is undeserved, and thus being deprived of it is not a punishment in the sense that it's a withholding of a free gift. And you can't just rely upon English words like "condemned" out there but need to cite the original languages, which are more precise. Now, the Jansenists condemned this view as Pelagian, but their condemnation was condemned, and effectively the Church taught that this was not Pelagianism.
I'm not too interested in debating this issue, as the Church has never condemned St. Augustine's position, merely exonerated that of St. Thomas as not being Pelagian, and tenable by Catholics. IMO, both positions are theologically tenable. I happen to agree with St. Thomas and the Greek Fathers on the matter.