Articles

The Importance and Limitations of Peer-Review

Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science. But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses.

Overview

When the statement is made that research is “peer-reviewed” this is usually meant to refer to the fact that it has been published in a peer-reviewed journal. Different scientific disciplines have different mechanisms for determining which journals are legitimately peer-reviewed. In medicine the National Library of Medicine (NLM) has rules for peer-review and they decide on a case by case basis which journals get their stamp of approval. Such journals are then listed as peer-reviewed.

The basic criterion is that there is a formalized process of peer-review prior to publication – so this presents a barrier to publication that acts as a quality control filter. Typically, the journal editor will give a submitted paper to a small number of qualified peers – recognized experts in the relevant field. The reviewers will then submit detailed criticism of the paper along with a recommendation to reject, accept with major revisions, accept with minor revisions, or accept as is. It is rare to get an acceptance as is on the first round.

The editor also reviews the paper, and may break a tie among the reviewers or add their own comments. The process, although at times painful, is quite useful in not only checking the quality of submitted work, but improving the quality. A reviewer, for example, may point out prior research the authors did not comment on, or may point our errors in the paper which can be fixed.

It is typical for authors to submit a paper to a prestigious journal first, and then if they get rejected to work their way down the food chain until they find a journal that will accept it. This does not always mean that the paper was of poorer quality – the most prestigious journals have tons of submissions and can pick and choose the most relevant or important studies. But sometimes it does mean the paper is mediocre or even poor.

The limitations of Peer-Review

It is important to realize that not all peer-reviewed journals are created equal. Small or obscure journals may follow the rules and gain recognized peer-reviewed status, but be desperate for submissions and have a low bar for acceptance. They also have a harder time getting world-class experts to review their submissions, and have to find reviewers that are also farther down the food chain. The bottom line is that when a study is touted as “peer-reviewed” you have to consider where it was reviewed and published.

Even at the best journals, the process is only as good as the editors and reviewers, who are people who make mistakes. A busy reviewer may give a cursory read through a paper that superficially looks good, but miss subtle mistakes. Or they may not take the time to chase down every reference, or check all the statistics. The process generally works, and is certainly better than having no quality control filter, but it is also no guarantee of correctness, or even the avoidance of mistakes.

Peer-reviewers also have biases. They may be prejudiced against studies that contradict their own research or their preferred beliefs. They may therefore bias the published studies in their favored direction, and may be loath to give a pass to a submission that would directly contradict something they have published. For this reasons editors often allow authors to request or recommend reviewers, or to request that certain people not be asked to be reviewers. Each journal has their own policy. Sometimes an editor will specifically use a reviewer that the authors request not be used, thinking they may be trying to avoid legitimate criticism.

The process can be quite messy, and full of politics. But in the end it more or less works. If an author thinks they were treated unfairly by one journal, they can always go to another or they can talk directly to the editor to appeal a decision and try to make their case.

Perhaps the biggest weakness of peer-review, however, is when an entire discipline of peers is lacking in some fundamental way. For example, there are now many journals dedicated to so-called “alternative medicine” (CAM).  The editors of such journals tend to have a pro-CAM bias, and they find reviewers with a pro-CAM bias. So pretty much any pro-CAM article can get published. Some have enough ideological friends at the NLM that they can get approved as peer-reviewed, despite glaring biases in their editorial policy.

Post Publication Peer Review

The term peer-review is sometimes used to refer to the fact that papers are read and reviewed by the broader scientific community once they are published. However, this post-production review should not be confused with “peer-reviewed” and that term should not be used to refer to post-publication review, to avoid confusion.

The process, however, is even more critical to quality control in science. Now, instead of one editor and 2-3 reviewers looking at a study, dozens or hundreds (maybe even thousands) of scientists can pick over a study, dissect the statistics and the claims, bring to bear knowledge from related areas or other research, and provide detailed criticism. This is the real “meat grinder” of science. Hundreds of reviewers are more likely to find problems than the few pre-publication reviewers. Arguments can be tested in the unforgiving arena of the scientific community, weeding out bad arguments, honing others, so that only the best survive.

Conclusion

Here is the bottom line – peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (15) ↓

15 thoughts on “The Importance and Limitations of Peer-Review

  1. superdave says:

    Even the big journals are not without problems. They try to publish the most intersting and cutting edge entries they can find and this leaves them more open to fraud than you might expect. Another problem, say a reviewer is reviewing a competitor with similar research, he may be tempted to lift ideas from that paper not yet in publication, or delay his own review process so his paper can beat out that paper in publication date.

  2. superdave says:

    However, it should be pointed out that in only one of the three cases I presented does fraudulent research actually get published.

  3. Joe says:

    Orac had an interesting post about an article favorable to homeopathy in the journal Chest. http://scienceblogs.com/insolence/2007/07/homeopathy_in_thecringeicu_1.php

  4. Oldfart says:

    Well, having read this article and superdave’s comment, I am much less reassured when comparing peer-reviewed based arguments with woo arguments since it seems obvious that even the peer-reviewed process cannot be ultimately trusted.

    As a layman, my problem is that I don’t have enough time or life to get a doctorate in everything…………or become tightly focused in ALL areas of science. I have to depend on something. I have to depend on expert testimony whether I am a common citizen, a governmental head, a political head, a military commander or a mother. And it is very difficult for the common person to argue against the woo that overwhelms us from every direction if the mainstay of arguments from science is faulty. Because it just leaves that little crack which the woo-believers pry open into a gaping hole.

  5. That is why you need replication and consensus – no single study, or small group of studies, is compelling by itself. But that does not mean that we cannot get to a reliable conclusion. It’s just harder than most people think.

  6. DavidCT says:

    Thanks for the review of the peer review process. A reminder of why if the information seems important, you have to read more than the abstract. Even with otherwise decent journals, the CAM sections will often will often use a very different set of “peers”. For some reason these journals continue to be well respected. Thinking is required for any source of information.

    I think I’ll just start a new career as a quack. This real science is way too much work.

  7. weing says:

    David,
    That’s right. Just get yourself one of those devices Harriet mentioned yesterday and you could be raking in an extra $50K per month.

  8. Jules says:

    I’m surprised you didn’t mention the impact factor, or services that allow you to see how many papers have cited a particular piece of research. In my mind, those are far more useful for assessing the validity of research than any sort of reputation that a journal can have.

  9. weing says:

    Jules,
    That can be manipulated too. You cite my paper, I’ll cite yours, and so forth, and pretty soon your impact factor and mine are way up there.

  10. daedalus2u says:

    There is no substitute for actually understanding the science. No amount or peer review can substitute for that.

    There was a book (published by the Nazis) titled ‘100 Authors Against Einstein’, telling how those authors thought that Relativity was wrong. Reportedly Einstein said, “If I were wrong, then one would have been enough!”

    Paradigm breaking science is extremely difficult for peers to evaluate. Thomas Kuhn talks about this.

    http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions

    Peers are good at evaluating the work-a-day science that the vast majority of peers do, what Kuhn calls “ordinary science”. They are not good at weeding out fraud, or in evaluating paradigm breaking science.

    I mostly don’t look at citation indexes or what journal the paper is in. If you can’t tell a good paper by reading it, then you need to get a lot more background in that field to be able to understand it. You need to be able to read bad papers and be able to tell why they are bad.

    Rather than hold up peer review as the be-all and end-all of scientific reliability, I think we should instead understand exactly and precisely what all scientific papers are, they are scientific communications from one scientist to another scientist. Scientific papers are not written for non-scientists to understand. They are written for specialist scientists in the relevant field to understand. If you don’t understand a paper, you need to read its references, and then the sub-references and then the sub-sub-references. You need to keep reading the literature until you do understand it. If you don’t understand a paper, you have no business citing it or writing about its data or conclusions as if you do.

    Orac recently wrote about someone (a non-scientist) who wrote some absolute nonsense blaming the histadine (0.78 mg) in a certain vaccine for life threatening complications. People like this didn’t arrive at their ideas through an understanding of the science behind them, pointing out how their non-scientific ideas are wrong isn’t going to get them to change their mind.

  11. ama says:

    Hi, all,

    this is not directly a comment to “peer-reviewed”. Instead it is a note about NOT peer-reviewed.

    http://www.salzburger-fenster.at/rubrik/lokales/2908/weiter-riesenwirbel-um-salzburger_9937.html

    http://www.onko-i.si/sl/mediji/novice/

    [*QUOTE*]
    ——————————————————–
    Novice

    Stališče Onkološkega inštituta Ljubljana do zdravljenja dr. Klehra
    29. 08. 2008

    V slovenski in avstrijski javnosti so se v zadnjih dneh ponovno odprla številna vprašanja, vezana na zdravljenje onkoloških bolnikov pri avstrijskem zdravniku dr. Nikolausu Klehru. Stališče Onkološkega inštituta, ki ga je strokovni svet OI sprejel že marca, in mnenje avstrijske zdravniške zbornice o delovanju imenovanega zdravnika, ki ga je na pobudo OI pridobila Zdravniška zbornica Slovenija, si lahko preberete v priponki.

    Mnenje strokovnega sveta OI (pdf, 1.25 Mb)

    Mnenje Avstrijske zdravniške zbornice (pdf, 862.95 Kb)

    Arhiv novic
    ——————————————————–
    [*/QUOTE*]

    In Europe Klehr, a known charlatan, did it again, and now in Slovenia the Onkološkega inštituta Ljubljana seems to make complaints about Klehr being NOT stopped by the Austrian authorities.

    In 2007 Klehr was in court in Germany. But as he refused to make statements, he was considered as a witness – and got away…

    The scandal is much larger than can be guessed at first sight. Today I got this tip:

    http://www.auravia.com/23.html

    The data from these web-sites was saved as pieces of proof:
    http://www.ariplex.com/ama/klehr_3.zip

    Please heat up the grill.

    Thank you,
    ama

  12. estherar says:

    As a reader of research, I’m sometimes amazed at what passes peer review. For example, somebody peer-reviewed the various Geier papers…

Comments are closed.