Some thoughts on a topic that has come to bear in my own life, and that is discussed in this thread:
First, if you buy a book second-hand (Goodwill, used bookstore, etc.), the author has already been paid for the book, so you do the author no injustice. Indeed, I have heard of people who do not wish to enrich the author of a bad book, but still need to use the book for legitimate research purposes or some similar reason, deliberately seeking to buy it second-hand, so they will not enrich the author.
Downloading the copy of something I already own, either first purchase or second-hand, also seems to pose no injustice. I am merely converting it into a format that is easier to use. For instance, we are reading Tolkien's The Hobbit in homeschool literature class. My son has a new paper copy from a retail bookstore, and I have a cheap second-hand paperback version for my own use. However, many evenings, I prefer to be able to read it on my computer, so I downloaded a free copy from a high school English honors program online. Tolkien's estate has suffered no injustice.
A book that is out of print also seems to be fair game for copying or downloading. If it is out of print, then the author has been enriched all that he is ever going to be enriched. If he wants more people to pay for rights to read the book, then he needs to arrange for more copies to be published.
Works in the public domain are entirely fair game.