Ordinarily we mean by the phrase, "God wishes all men to be saved," all men capable of moral acts. An unbaptized infant is incapable of performing a moral act, because although he has reason, he does not have the use of his reason. When we say that God gives to all men sufficient grace to save their souls, we are of course referring to those capable of producing moral acts.
We know that the unbaptized infants do not suffer torments for evil acts that they obviously did not commit. As to the fate of the unbaptized infants, the Church has said very little. Why some infants die with baptism and others do not is a mystery.