who can doubt that nonbaptized infants, having only original sin and no burden of personal sins, will suffer the lightest condemnation of all? I cannot define the amount and kind of their punishment, but I dare not say it were better for them never to have existed than to exist there. But you, also, who contend they are, as it were, free of any condemnation, do not wish to think about the condemnation by which you punish them by estranging from the life of God and from the kingdom of God
He distinguishes his position from those who say the infants are "free of any condemnation" and proceeds to elaborate that they admit a single condemnation, the deprivation of the Beatific Vision that is.
Clearly, Augustine believed infants suffered some kind of punishment besides the "punishment"
(improperly so called because not receiving an inheritance is not a punishment) of deprivation of Heaven his opponents held.