Check Yourself

This rather long piece at Outside the Beltway delves into an interesting fact about a lot of scientific studies that get published: They're probably wrong.

The whole thing should be read, and it would be worthwhile to take a look at the original piece from The Economist that the Beltway article references. The issue is not that the scientific method itself is flawed. It isn't -- the process of predicting or guessing results and performing experiments which either confirm or refute the predictions remains one of the best methods humanity has ever devised for finding out how things work.

Nope, the problem is not with the method of investigation, but more with the method of disseminating its results. Regular human failings like confirmation bias are joined by professional pressure to publish, a preference for data and results that show something new rather than confirm something already known and over-reliance on fallible peer review to weed out error.

Both pieces highlight a 1998 study which showed that people about to take an intelligence test did better when they imagined a college professor than when they imagined a soccer hooligan. In April 2013, a journal article reported that nine separate experiments failed to reproduce the results claimed by the 1998 researchers. Which to me only makes sense. The average soccer hooligan has to keep a raft of team statistics straight in his head while slamming it against other hooligans' fists, while the average professor just has to slam the door shut on any challenge to his or her worldview once acquiring tenure.

Of course that's a ridiculous rationale, but the idea itself is ridiculous enough that some of those nine experiments should have been done back in 1998. Both the Beltway and Economist articles point out some of the reasons they weren't, which help lead to the problem they're talking about. Although any truly innovative result should engender skepticism not only among scientists but also the press that reports on their work, the combination of a need to grab eyeballs and the absence of any significant cost of getting something wrong only make a bad problem worse once it hits ink, digital or otherwise.

I am not entirely sure what I should do with this newfound reason to be skeptical about the results of scientific studies. Right now I'm leaning towards eating Twinkies, French fries and Quarter Pounders with cheese for the rest of my life, but I'm still open to how often I should replace the above menu with pizza and chicken fried steak with gravy.