Open-Access Peer Review: Increasing the Noise To Signal Ratio

Readers of Science Based Medicine are quite familiar with the distressingly common logical leap made by disgruntled healthcare consumers into alternative medicine. It goes something like this: I had a terrible experience with a doctor who [ignored/patronized/misdiagnosed] me and I also heard something horrible in the media about a pharmaceutical company’s misbehavior [hiding negative results/overstating efficacy/overcharging for medications], therefore alternative treatments [homeopathy/acupuncture/energy healing, etc.] must be more effective than traditional medicine.

Much to my dismay, a similar logical leap is being made about online health information. It goes something like this: Peer reviewing is biased and often keeps innovative research hidden to the world at large, therefore the best kind of peer review is open-access where anyone in the world can contribute.

You may feel free to slap your forehead now.

While I have absolutely no doubt that doctors have their shortcomings, and that some have created less than pleasant healthcare experiences for their patients – the solution to these shortcomings is not to dive headlong into snake oil. Moreover, I agree that the current peer-review process has its flaws and limitations – the solution is not to ask Aunt Enid in Omaha what she thinks of the recent meta-analysis of perioperative beta blockers in patients having non-cardiac surgery.

Peter Frishauf, the founder of Medscape, recently published a webcast editorial predicting that:

“Peer review as we know it will disappear. Rather than the secretive prepublication review process followed by most publishers today, including Medscape, most peer review will occur transparently, and after publication.”

He goes on to describe a Wikipedia-like review scenario where:

“Any user can start an article, link it to related sources, and publish revisions with a click of the mouse. Anyone who reads an article can edit it.”

I know and like Peter very much, and his foresight (that publishing should become open-access), combined with the leadership of editorial heavy-weight, Dr. George Lundberg, led to the creation of the first really successful, quality, free online medical journal. This was no small feat, and a sure victory for global medical education efforts.

But the reason for The Medscape Journal’s success is not the “democratization” of peer review – but the democratization of access to trustworthy information. The quality controls are still in place – and must remain so – otherwise its value as a peer-reviewed journal will be utterly lost. Who should trust the edits of unqualified readers? Should science be determined by popular vote? Should all research be published by journals, regardless of its fatal flaws?

I wrestle daily with people who truly believe that “the wisdom of the crowds” will revolutionize medical information. These people are part of a “do it yourself” healthcare movement – spurred on by frustration from a broken system, overconfidence in their ability to interpret complex data, and an ignorance of the potential biases, statistical waffling, and methodologic flaws that so easily mislead even the most savvy reviewers. They like to imagine themselves as hip, anti-establishment pioneers – daring to “push the envelope” in the face of paternalistic scientists who cling tenaciously to an unreasonable status quo. In reality, however, they’re working to unravel the progress we’ve made over a century of painstakingly controlled scientific experiments – systematically designed to remove bias, errors, and false assumptions wherever possible and uncover the raw truth of things. As David Gorski (and The Onion) has pointed out, science is hard.

Now let me explain why open-source peer review does not solve the current publishing “bias problem” and why it would, in fact, be inferior to our current system. Advocates of various Web 2.0 tools argue that flinging wide the doors to all readers (à la Wikipedia) is the single best way to clarify, edit, and correct information. I can see how this may be useful for historical documents, but scientific literature is another matter altogether. Not only are unqualified reviewers likely to do more harm than good with their well-intentioned edits, but the fact of the matter is that there is little incentive for most people to do any reviewing at all.

Slate recently discovered (quite accidentally) this timeless truth: people are unlikely to participate in activities that have no personal value to them. Their review of physician-rating sites revealed dismal participation rates. The site creators believed that scads of patients would immediately descend upon their open-source tools in order to selflessly fill out long questionnaires about the quality of their physician interactions. Alas, the “revolutionary” and empowering rating tools that promised to help patients find out the “truth” about their doctors’ quality were met with a yawn and the sound of crickets. The few ratings that did exist (one site had 137,832 listed doctors and 5,709 patient reviews) tended to skew towards negative experiences, completed by patients whose frustration had reached sufficient levels to carry them through the arduous rating process.

And so, since good peer review of the medical literature is far more arduous than even physician ratings, those who choose to participate in an open source scenario are likely to have an axe to grind. The potential for bias online probably exceeds bias in the current peer review process, where reviewers are hand-selected for subject matter expertise from institutions where there is no suspected conflict of interest, and both authors and reviewers are purposefully blinded so that pre-conceived notions of individual or institutional competencies are less likely to play a role. The fact that the reviewer names are never revealed further reduces incentive to “make a name for oneself” through academic posturing of a self-aggrandizing nature.

Suggesting that the current peer review process should be replaced with an open-source free-for-all is kind of like suggesting that double-blind, placebo-controlled trials should be substituted with unblinded trials where subjects and investigators both know if they’re getting a placebo or the treatment under investigation. Wouldn’t it be fairer for everyone to just have open access to all that information? Isn’t it unnecessarily secretive and paternalistic to hide the “truth” from study subjects? Maybe a better way to figure out if a drug really works is to get all the patients together to brainstorm about their experiences? Poppy cock!

Now, I agree that we could improve our current peer review process by vastly expanding the reviewer pool (this would also help to speed up the process of getting good research published and available for public consumption). But that pool must still consist of reviewers with real expertise – and the motivation problem must be solved. What’s in it for the reviewer? Well, why not devise a system whereby CME credit is offered to qualified physicians who wish to participate in peer review? Why not allow online journals to pull from a large, shared pool of carefully vetted but anonymous review volunteers?

There are ways to improve the scientific peer review process, and there are ways to increase the noise to signal ratio. Open-access peer review is certainly an example of the latter.

Posted in: Science and Medicine, Science and the Media

Leave a Comment (10) ↓

10 thoughts on “Open-Access Peer Review: Increasing the Noise To Signal Ratio

  1. overshoot says:

    The motivation problem is (IMHO) the killer.

    Peer review extends far beyond the realm of scientific publishing; it is, according to mountains of research, the most powerful up-front quality tool we have. Every instrument you use, every chip in your computer, etc. all have “design review” steps built into the flow from beginning to end.

    Unfortunately, those reviews are all too often cursory. “Yup, that looks like a whatsis” is about all you get [1]. A good review is work, it takes time, and unfortunately time is never a surplus commodity. Especially for those whose attentions are most worth having. Reviewers, after all, have their own deliverables.

    My best experiences with industrial peer review, oddly enough, are in the standards-setting context. Really good people are assigned to work on committees, and the quality of the standards bears directly on their other deliverables — including their employers’ market success. We got great, aggressive, meticulous review work from multiple sources.

    My proposed solution in industry is to have people assigned full-time to doing review and nothing but. IMHO it’s a great job for good junior people; it’s educational as all get out and exposes them to a great wealth of material that they would otherwise take years to see.

    In academia, the “service” part of the workload would serve, always assuming that there were controls in place to ensure actual attention to the task (much as with teaching, actually.)

    Medicine? Interesting question. Your suggestion of CMEs might be a good starting place, with the same caveats as above.

    [1] Says /me, taking a break from spending a week preparing for a design review. I doubt that any of the reviewers will read the material before dropping in for coffee at the meeting.

  2. Karl Withakay says:

    >>>Well, why not devise a system whereby CME credit is offered to qualified physicians who wish to participate in peer review?

    That’s one of the best ideas I’ve heard all week. To continue the thought, consider extending the concept to the academic/PHD world, use it as criteria for granting tenure.

    “Hmmm, you’ve got a good publication record, but you don’t seem to be on any peer review committees; we like our tenure candidates to contribute on both sides of the publication process.”

  3. Karl Withakay says:


    People not in the medical or legal industry may not realize the significance of Continuing Medical Education (CME). I believe it’s generally a requirement for maintaining you license status.

  4. hatch_xanadu says:

    This is fantastic. Thank you.

  5. Karl Withakay says:

    Edit: I meant to either edit out mention of the legal industry, or add reference to CLE (Continuing Legal Education) and did neither.

  6. DLC says:

    Of course an “open to the public” review system is foolish.
    You have to actually be a peer to write a peer review.
    Joe off the street isn’t qualified to comment on advanced medical (or other scientific) research, any more than he is qualified to fly a 747. (even though modern 747s all but fly themselves.)

  7. Joe says:

    The only thing The Wikipedia is useful for is looking up references to reliable literature, which one can then consult. It was most useful when I wanted to know what chiros are saying, today, about Innate Intelligence.

    Wiki’s “Neutral Point of View” (NPOV) requirement is nonsense; yet, Frishauf seems to think it is good. Is referee bias that much of a problem in medical literature? Moreover, NPOV cannot be truly implemented. For example, if one writes “naturopathy is a medical profession” Wiki accepts it; whereas, “naturopathy is not a medical profession” (which is true) is not acceptable. I know because they called me a vandal for inserting the negative. A couple months later, all comparisons to medicine were gone from the introduction.

    I would never contribute anything to an open-source site. After going to the trouble of composing a piece, one can find it completely replaced- time wasted.

    Steve Novella is considering a science wiki, about which I am enthusiastic; I assume the articles would only come from, and be edited by, a select group.

    It seems to me that Medscape already has it right. Don’t they allow comments on articles?

    CME credit is a great idea.

  8. MKandefer says:

    PLoS ONE ( is an open journal, in the sense that there is still an editorial board that distributes submitted papers for peer-review, but it also allows comments to be made on articles from registered members. I’m not certain how well this works out, but it seems like an interesting prospect. It’s not a wiki though, where anyone has access to the works submitted.

  9. Joe says:

    @MKandefer, you are right. The distinction is “open-source” as opposed to free. Open-source allows one to alter the primary article. That is unacceptable in science, as opposed to The Wiki; which is what renders The Wiki useless for scholarly publication.

Comments are closed.