Readers of Science Based Medicine are quite familiar with the distressingly common logical leap made by disgruntled healthcare consumers into alternative medicine. It goes something like this: I had a terrible experience with a doctor who [ignored/patronized/misdiagnosed] me and I also heard something horrible in the media about a pharmaceutical company’s misbehavior [hiding negative results/overstating efficacy/overcharging for medications], therefore alternative treatments [homeopathy/acupuncture/energy healing, etc.] must be more effective than traditional medicine.
Much to my dismay, a similar logical leap is being made about online health information. It goes something like this: Peer reviewing is biased and often keeps innovative research hidden to the world at large, therefore the best kind of peer review is open-access where anyone in the world can contribute.
You may feel free to slap your forehead now.
While I have absolutely no doubt that doctors have their shortcomings, and that some have created less than pleasant healthcare experiences for their patients – the solution to these shortcomings is not to dive headlong into snake oil. Moreover, I agree that the current peer-review process has its flaws and limitations – the solution is not to ask Aunt Enid in Omaha what she thinks of the recent meta-analysis of perioperative beta blockers in patients having non-cardiac surgery.
Peter Frishauf, the founder of Medscape, recently published a webcast editorial predicting that:
“Peer review as we know it will disappear. Rather than the secretive prepublication review process followed by most publishers today, including Medscape, most peer review will occur transparently, and after publication.”
He goes on to describe a Wikipedia-like review scenario where:
“Any user can start an article, link it to related sources, and publish revisions with a click of the mouse. Anyone who reads an article can edit it.”
I know and like Peter very much, and his foresight (that publishing should become open-access), combined with the leadership of editorial heavy-weight, Dr. George Lundberg, led to the creation of the first really successful, quality, free online medical journal. This was no small feat, and a sure victory for global medical education efforts.
But the reason for The Medscape Journal’s success is not the “democratization” of peer review – but the democratization of access to trustworthy information. The quality controls are still in place – and must remain so – otherwise its value as a peer-reviewed journal will be utterly lost. Who should trust the edits of unqualified readers? Should science be determined by popular vote? Should all research be published by journals, regardless of its fatal flaws?
I wrestle daily with people who truly believe that “the wisdom of the crowds” will revolutionize medical information. These people are part of a “do it yourself” healthcare movement – spurred on by frustration from a broken system, overconfidence in their ability to interpret complex data, and an ignorance of the potential biases, statistical waffling, and methodologic flaws that so easily mislead even the most savvy reviewers. They like to imagine themselves as hip, anti-establishment pioneers – daring to “push the envelope” in the face of paternalistic scientists who cling tenaciously to an unreasonable status quo. In reality, however, they’re working to unravel the progress we’ve made over a century of painstakingly controlled scientific experiments – systematically designed to remove bias, errors, and false assumptions wherever possible and uncover the raw truth of things. As David Gorski (and The Onion) has pointed out, science is hard.
Now let me explain why open-source peer review does not solve the current publishing “bias problem” and why it would, in fact, be inferior to our current system. Advocates of various Web 2.0 tools argue that flinging wide the doors to all readers (à la Wikipedia) is the single best way to clarify, edit, and correct information. I can see how this may be useful for historical documents, but scientific literature is another matter altogether. Not only are unqualified reviewers likely to do more harm than good with their well-intentioned edits, but the fact of the matter is that there is little incentive for most people to do any reviewing at all.
Slate recently discovered (quite accidentally) this timeless truth: people are unlikely to participate in activities that have no personal value to them. Their review of physician-rating sites revealed dismal participation rates. The site creators believed that scads of patients would immediately descend upon their open-source tools in order to selflessly fill out long questionnaires about the quality of their physician interactions. Alas, the “revolutionary” and empowering rating tools that promised to help patients find out the “truth” about their doctors’ quality were met with a yawn and the sound of crickets. The few ratings that did exist (one site had 137,832 listed doctors and 5,709 patient reviews) tended to skew towards negative experiences, completed by patients whose frustration had reached sufficient levels to carry them through the arduous rating process.
And so, since good peer review of the medical literature is far more arduous than even physician ratings, those who choose to participate in an open source scenario are likely to have an axe to grind. The potential for bias online probably exceeds bias in the current peer review process, where reviewers are hand-selected for subject matter expertise from institutions where there is no suspected conflict of interest, and both authors and reviewers are purposefully blinded so that pre-conceived notions of individual or institutional competencies are less likely to play a role. The fact that the reviewer names are never revealed further reduces incentive to “make a name for oneself” through academic posturing of a self-aggrandizing nature.
Suggesting that the current peer review process should be replaced with an open-source free-for-all is kind of like suggesting that double-blind, placebo-controlled trials should be substituted with unblinded trials where subjects and investigators both know if they’re getting a placebo or the treatment under investigation. Wouldn’t it be fairer for everyone to just have open access to all that information? Isn’t it unnecessarily secretive and paternalistic to hide the “truth” from study subjects? Maybe a better way to figure out if a drug really works is to get all the patients together to brainstorm about their experiences? Poppy cock!
Now, I agree that we could improve our current peer review process by vastly expanding the reviewer pool (this would also help to speed up the process of getting good research published and available for public consumption). But that pool must still consist of reviewers with real expertise – and the motivation problem must be solved. What’s in it for the reviewer? Well, why not devise a system whereby CME credit is offered to qualified physicians who wish to participate in peer review? Why not allow online journals to pull from a large, shared pool of carefully vetted but anonymous review volunteers?
There are ways to improve the scientific peer review process, and there are ways to increase the noise to signal ratio. Open-access peer review is certainly an example of the latter.