Nancy Pearl, author of Book Lust, has a suggestion about when you should decide if the book you are reading is worth it (quoted here at Austin Kleon's Tumblr). The idea is to not waste the time you are on this earth by reading lousy books any longer than you have to.
If you are 50 years old or younger, Pearl says, you should give a book 50 pages to prove itself, and abandon it at that point if it hasn't. If you are older than 50, you should subtract your age from 100 and take that number as the number of pages you should read before moving on.
Pearl is a clever writer and her book has some great lists of suggestions, and I imagine at least some of her suggested limit is tongue in cheek. After all, this would mean that nonagenarians would probably not finish very many books at all, and how much fun would it really be to have read ten pages of a hundred books instead of all of one or two?
Plus, there are many books that start out slowly and you may be well into them before you realize they have taken root. Patrick O'Brian's Master and Commander, the first of his famed Aubrey-Maturin series, is written with a very definite 18th century feel and took me no little time to acclimatize myself to. Had I given up according to Pearl's rule, I would never have read the entire series and thus would have deprived myself of some of the most fun I have ever had reading anything. And I would have never encountered the line, "Jack, you have debauched my sloth!" and that would be a tragedy of cosmic proportions.
Plus, there are many books that start out very well but die after the 50-page mark -- sometimes all the way at the very end. Stephen King has this problem. A lot of his books sink hooks into readers and carry them along until the very end, at which time he proves himself unable to bring a payoff appropriate to the story. He isn't the only one, and you usually have to go waaaaay beyond 50 pages before you find that out.
And then there are some books you should dismiss without scanning a single page -- memoirs by politicians or anyone under 30 and just about anything with James Patterson as a co-author would be good examples.