Let's say you were Baptized as an infant, but you didn't come to believe until you were, for example, 20 years old. Would you pay back the sins you committed before you came to believe?

Assuming we died in the state of sanctifying grace, we will be cleansed i.e. purified by the fires of Purgatory till amends are made for whatever traces of sins we have on our souls at the time of our death, that though they were forgiven in the sacrament of Penance, we did not make sufficient penance while on earth to completely eradicate them from our souls.

....the fire of purgatory, in which the souls of just men are cleansed by a temporary punishment, in order to be admitted into their eternal country, into which nothing defiled entereth.

One sure way of avoiding purgatory is to die after receiving the sacrament of Extreme Unction with the proper disposition. (see my signature)

Just as God's mercy is infinite so also is his justice. If the person sinned through ignorance then it would seem that his punishment would be less than if he knew better. Only God knows.
It is best to try to make up for our sins in this life than to have to suffer for them in the next where Jesus said you will not be released until the last penny has been paid.