The piece that was originally here has been moved. (The comments remain attached to the present post.) It was a piece produced for amusement that mined the Amazon reviews for a particular book in search of evidence that Amazon had censored one-star reviews. It turned out that the evidence didn't stand up. The piece originally stated that the evidence might not hold up and pointed out the reason it might not, which did in fact turn out to be the case. The piece got a lot of attention anyway. At this point I'd rather have it elsewhere than on my blog.
However, based on the (limited and anecdotal) evidence I've seen, I *think* Amazon may have guidelines in place that make it harder for negative reviews to get posted than positive ones, even though those guidelines probably aren't "don't post negative reviews." Rather, negative reviews may to have more hoops to jump through. See Brian Powell's experience for more.
Thank you for writing this. I too have long taken Amazon's reviews very seriously in making purchasing decisions. I'm appalled and I feel like Amazon has stolen from me. I plan to get the word out until this stops!
Posted by: Staci | March 31, 2006 at 11:57 AM
You fail to take into account the "was this review helpful" system.
I am guessing that reviews which are consistently rated as being helpful get more weight, with the amount of additional weighting derived from the number and percentage of people who found it hlepful.
Posted by: Adam | March 31, 2006 at 12:24 PM
"But let's assume that Amazon would claim nothing is wrong: there is no underlying pattern that might lead to a higher propensity of one-star reviews since yesterday. That would imply that the any special preponderance of one-star reviews since yesterday would be a mere coincidence. So let's assume it is, and see where that takes us."
Er... or else it is because people read the same reddit article as you, and are now testing whether the claims are true.
Posted by: Anomolous | March 31, 2006 at 12:26 PM
all reviews, negative and positive, plus 10% off amazon in the books department.
Posted by: Smak | March 31, 2006 at 12:29 PM
Er... I KNOW. That's why I said "In addition, there may be other explanations to the propensity of recent one-star reviews. For instance, they could all be due to the reddit posting. However, reading the reviews, this does not appear to be the case. Most of them give ample evidence of direct knowledge of the book and thoughtful criticism."
As I also said, the calc is "for fun". Let's not take it too seriously.
Maybe I'll have to give an ever stronger caveat.
Posted by: Gary Robinson | March 31, 2006 at 12:30 PM
It's time for the public to be more exacting on the quality of rating system, e.g. by demanding average and standard deviation to come along with the valuation, see my comment in reddit.
Posted by: alv | March 31, 2006 at 12:39 PM
"I understand that Amazon has to round the number because they show the review as a graphic, but they do round to half-stars, so why are they rounding 3.3 up to 4? Something seems fishy here."
(18*1 + 5*2 + 3*0 + 4*4 + 35*5) / (18+5+0+4+35) = 3.53225806451613
Posted by: Michiel | March 31, 2006 at 12:50 PM
Wouldn't this type of analysis be a lot more interesting for a book randomly selected on Amazon, rather than one in which obvious biases are at play right now?
I understand that this is just for your own amusement, but doesn't this one simple change vastly improve the quality of your "study" and therefore your post?
Posted by: Josh Staiger | March 31, 2006 at 01:34 PM
We know that this book has been the subject of one person (Brian) getting his one-star reviews rejected, and the reviews that were rejected look perfectly good to me. So, it seems arguable that a valid experiment is to test the reviews for just one book, which is one that is suspect. Because, even if every book doesn't receive this kind of tilt, if some subset of the books do, that's a problem.
So a response would be: I should do as you propose if the test was to see if as a uniform principle, reviews were screened in this way. But I am less interested in that than I am in the hypothesis that SOME books are screened that way.
Note that I did NOT use the data that brought the book to my attention in the analysis. So the analysis was not skewed by that data. If indeed Amazon does not ever have a skew against any book, then it wouldn't matter that the book was brought to my attention by someone who thought his review was unfairly rejected, as long as I didn't use that data in my analysis.
Posted by: Gary Robinson | March 31, 2006 at 01:37 PM
This is an interesting case about Amazon. But also it shows yet another reason why reddit sucks. The author has to rewrite his commentary because of the influence of the reddit engine. If this is "Web2.0", then I'm on a mission to discover "Web3.0", thank you!
Posted by: vanjulio | March 31, 2006 at 01:47 PM
Hey, I think the comments I've received have helped make something that was a lot better than the original. Some of the criticism is valid, and I've updated the piece due to it, and with the updates, it's a more useful piece. I don't see why you would think that was bad...
Posted by: Gary Robinson | March 31, 2006 at 01:50 PM
Erm,
If you read the latest one star reviews you will see that they are the original critical reviews from Brian Powell posted under other Amazon user's names. Good fun!
There is a serious point - is it in Amazon's interest to leave up negative reviews? Well sensible folks would say yes - 'cos if I lose faith in the reviews I will lose faith in Amazon. But - I have too often purchased books where in retrosepect it looks like the review was taken from the dust jacket and was not based on any critical reading.
Posted by: Mike Griffiths | March 31, 2006 at 02:02 PM
It should be pointed out, as I wrote on Reddit, that Powell's reviews were very poor:
--------------------
One of Amazon's replies includes this:
That is an accurate assessment. I read Powell's criticisms carefully, and I was left basically uninformed.
The book is never quoted. Specific examples of what the book states and why it is wrong are not given.
All of his criticisms boil down to saying that the book is wrong because its author does not understand physics. As proof, Powell gives his credentials - "physics graduate students and physics educator" - but never gives a specific example of an error in the text.
Why? How? He never explains. I agree with Amazon on this one.
Posted by: Adrian | March 31, 2006 at 03:13 PM
Thanks for looking into this situation. I just wanted to make clear that all of this happened just about a year ago (April 05). I had simply given up and had not thought about any of this until my link showed up on reddit. I understand that today there are many negative reviews, however, this was not the case for a *very* long time. In fact, I find it very suspicious that Amazon has begun posting reviews all of a sudden just as this news hits.
Posted by: Brian Powell | March 31, 2006 at 03:14 PM
Brian: thanks for your feedback. Particularly where you say "I find it very suspicious that Amazon has begun posting reviews all of a sudden just as this news hits." I agree, and that is noted in my piece.
It may indeed be that that is the sign of trouble -- the simple fact that negative reviews weren't getting through, and now, right after the reddit piece, they are. In which case the statistical analysis is a sign that that is happening.
Posted by: Gary Robinson | March 31, 2006 at 03:18 PM
11 reviews in two days is an unusually high number when compared to other books of similar sales rank.
Given that those 11 reviews all appeared after the reddit posting, I don't think it's entirely out of the realm of possibility to assume that *all* of them were due to the reddit effect.
*That* would be my most pessimistic case, not the 2 that you cite, in which case the analysis cannot be performed at all and the study becomes completely invalid.
Again, I understand this is all just for fun, but I think you have to at least concede that possibility.
Posted by: Josh Staiger | March 31, 2006 at 03:27 PM
Josh: it just seems so unlikely, though, if you read the reviews. I don't think the ones who were specific and give evidence of having read the book actually went out and read it due to reddit, OR that they were lying. Another possibility is that they had already read the book, and reddit convinced them to post a review. That too seems to me to be very unlikely.
So yes, I concede your point, but I still think that if we want to use the data to judge whether Amazon is tilted against one-star reviews, it's more reasonable to say that those possibilities are too unlikely to matter much.
Particularly since, weighed against those arguments that tell us to give the benefit of the doubt to Amazon, there is Brian's actual experience where they DID repeatedly reject his one-star review. If we're going to weigh in the pro-amazon arguments, shouldn't we weight that in too? But we can't. I think the approach I have taken is reasonable as a middle ground.
As to Adrian's comment above, I strongly disagree. I see positive reviews that seem to have the same problems and they aren't censored. Plus brian gave a couple of versions in response to the criticisms that seemed, to my mind, to address them.
As a reader, I WANT Brian's opinion, since he's grad student in physics. I don't need to see everything spelled out in detail. It's useful to get his opinion added to the mix either way. Requiring critics to give detailed criticism (while not doing that for fawning positive reviewers) slants the reviews toward the positive, and makes it harder for me as a consumer to make a reliable choice.
Because the NUMBERS on one side or the other due count, regardless of the logic of the arguments made.
Posted by: Gary Robinson | March 31, 2006 at 03:39 PM
My stats skills are rusty, but I'd also say that a sample size of two in your most pessimistic case is not large enough to say much of anything statistically significant, really...
Posted by: Josh Staiger | March 31, 2006 at 03:54 PM
Usually in statistics a sample size of 2 is not enough to generate statistical significance. For instance, if you want to know whether a coin is weighted toward giving heads and you throw it twice and get heads both times, the probability of that is .25.
The cutoff for something to be called "statistically significant" is usually .05 (see for instance http://www.surveysystem.com/signif.htm). This is usually called the "alpha-level"
The die example above therefore does not reach statistical significance. In most real-world cases, a small handful of cases do not result in statistical significance. So there's a vague sense in the common wisdom that you need more cases for significance.
But actually, the question is whether the alpha-level of .05 is attained or not, and it doesn't matter what the sample size is if it is attained.
But I reiterate what I said before. This is NOT a rigorous finding, for reasons discussed in our earlier exchange.
Posted by: Gary Robinson | March 31, 2006 at 04:08 PM
(As a side note, you need a sample size of 5 before you can reach statistical significance in flipping a coin.)
Posted by: Gary Robinson | March 31, 2006 at 04:20 PM
For pity sakes folks, are you telling me that with tens of thousands of books, many with thens or hundreds of reviews, that Amazon is going to detsroy its major marketing tool by censoring unfavorable reviewers?
Isn't it more likely that the author complained about Powell's reviews several times? Wouldn't the author have more reason to eliminate unfavorable reviews than the folish assumption that Amazon is actively monitoring a book with a limited audience and putting the integrity of their whole review structure in jeopardy?
It simply doesn't make sense.
It takes some big leaps to conclude that Amazon is doing something underhanded rather than responding to an authors complaints. One has to simply look at reveiws on any given book to know that there is little to no chance that Amazon is censoring them to increase their bottom line.
Get a grip folks.
Posted by: Clarke Green | March 31, 2006 at 04:38 PM
"Wouldn't the author have more reason to eliminate unfavorable reviews than the folish assumption that Amazon is actively monitoring a book with a limited audience and putting the integrity of their whole review structure in jeopardy?"
See this from a top 100 reviewer: "Amazon's "reviews" are screened for pertinence to the book, for duplication, and for various kinds of nastiness, including personal attacks. I can wax vitriolic when I really loathe a book, and I've written a few reviews that didn't make it past the censors. " (http://www.dancingbadger.com/amareview.htm)
Posted by: Gary Robinson | March 31, 2006 at 04:49 PM
Clarke -
Why does it matter who removes them??
If Amazon claims that their review section is honest (as they do) then how can they allow authors to remove negative reviews? They deny such accusations. If they are being honest, why do so?
If all authors where like McCutcheon, then there would only be 5-star reviews on amazon.com. How useful would the review service be then??
Posted by: Brian Powell | March 31, 2006 at 07:15 PM
Some time ago I developed the theory that these type of online reviews were mostly made by mad people. As one of your postings point out star ratings (and not just amazon) tend to be bipolar.I suppose that some are made by people with a grudge and that the others by people who don't get out much.
Posted by: Bob Galway | March 31, 2006 at 07:32 PM
I think it's also worth noting that any 'statistical analysis' based on a single book on Amazon is pretty much meaningless. How many books do they sell? How can you consider an analysis of reviews of one book to be statistically significant? If you want to make them look bad, I'm sure you can find a better example. If you want to make them look good, I'm sure you can find a better example. You've proved precisely nothing.
Posted by: Mark | March 31, 2006 at 11:29 PM