Scientists Assessing Scientists: Does Peer Review Work?

“And this is why they say that no one has ever seen a water-baby. For my part, I believe that the naturalists get dozens of them when they are out dredging; but they say nothing about them, and throw them overboard again, for fear of spoiling their theories.” The Water-Babies by Charles Kingsley, 1863, p. 89.

This quote from an old children’s book makes an interesting point. If a scientific paper flies in the face of an established theory, will it be thrown overboard? According to Richard Smith, MD, former editor of the British Medical Journal, it might. He says that peer review, the process whereby the community of scientists assesses which papers are worth publishing, who should get tenure, which projects should be funded, and even who should be awarded a Nobel prize, does not work. In fact, “we have no convincing evidence of its benefits but a lot of evidence of its flaws.”

Surely not! After all, the main argument of the folks wishing to stifle debate about global climate change, evolution, and the latest miracle drug is that challenges to their point of view cannot be found in peer-reviewed scientific journals. (Please note that this claim is false; there are a small number of papers in the peer-reviewed literature that challenge anthropogenic climate change, aspects of evolution, and other controversial theories.) But, according to Dr. Smith and the research he cites, peer review also does not “ensure quality of biomedical research,” “many studies published in medical journals…are not only scientifically poor but also have done great damage,” and put quite simply, “peer review doesn’t work.”

Interestingly, Dr. Smith says that peer review does a lot of harm. Quite apart from giving some theories more credibility than they merit, it is “an ineffective, slow, expensive, biased, inefficient, anti-innovatory, and easily abused lottery.” That is, authors from less prestigious institutions, those with new and innovative ideas, and those who challenge the status quo find that they cannot get published. And those who know how to work the system, say the right things, or have the right connections can. The icing on the cake is then when other scientists refuse to consider the theories of colleagues that think outside the box because those ideas are as yet unpublished.

So, what is the answer? Dr. Smith suggests that all papers should be published on the Internet so that 100′s instead of a few people can review them. Obviously, this system may cause information overload. In addition, it will not help the average person to figure out the scientific merit of what they are being told. That is the mission of AITSE: to educate to public so that they can assess the validity of “scientific” claims and to encourage integrity in science.